|
From: Peter S. <co...@pe...> - 2007-10-30 15:25:48
|
Hi Roberto, I think we should split the project in two phases: Phase One: Media upload mechanisms and media file download (with correct mime types etc) - but no media player. So the play button would go directly to the multimediaservlet. Phase Two: Embedded Mediaplayer using e.g. FMJ. About the file sys: Splitting on date folders is good (to speed of access), but in case of massive uploads, it might be better to split on MetaDataLogEntry id ranges. Maybe the folder structure should reflect this - e.g. 1000 files per folder (this could be customizable). I see the challenges regarding separating the project, and the challenge regarding different data types we've also had with other data types. The MetaDataLogEntry table should maybe have one field per datatype such as with the numericvalue field - that we don't use today - but maybe we should. regards, Peter On 10/30/07, Roberto MIlev <rob...@gm...> wrote: > > Hi Peter, > > Please find attached my ideas on on implementing the new feature. My > initial rough estimate for implementation is around 80 man hours. > > Please give me your thoughts. > > Regards, > Roberto > > Peter Salomonsen wrote: > > Hi Roberto - > > > > how is it going with the interceptor stuff - did you get to understand > > how it works? > > > > I have a new project proposal for you. We'd like to add multimedia > > capabilities to the MetaLogViewer - so that you can attach multimedia > > clips to a MetaDataLogEntry. The idea will be to create new columns > > that contains multimedia file names, and with a possibility to launch > > a player to view /play. Embedded players would be very nice - however > > supporting all the varieties of file formats may be difficult. Have to > > consider whether to stick to one format ( e.g. ogg/vorbis/theora - and > > use e.g. the flumotion cortado applet), or support multiple formats, > > and test frameworks such as fmj (which is an open source impl of java > > media framework). > > > > I don't think the media content itself should be stored in the > > MetaDataLogEntry table, but the file name could be, and there should > > be a file structure for easy retrieval. > > > > Get the idea? > > > > This makes the lmappletserver-addons.jar even more massive, and the > > question is if we could create a new Multimedia-extensions module? > > Should this even be a whole new separate project? > > > > Is this a project for you? In that case some sketches and models would > > be nice, would you have a look at it, and think of a solution proposal? > > > > regards, > > > > Peter > > > > On 10/23/07, *Roberto Milev* <rob...@gm... > > <mailto:rob...@gm...>> wrote: > > > > Hi Peter, > > > > I changed and commiter orm.xml. I will be looking at the > > interceptor stuff next. I will keep you posted. > > > > Regards, > > Roberto > > > > > > On 10/23/07, * Peter Salomonsen* <co...@pe... > > <mailto:co...@pe...>> wrote: > > > > Hi Robert, > > > > I've attached a metaloginterceptors property file example. > > This is a feature me and Jon has been working on, and the > > purpose is to be able to intercept calls within the > > lmappletserver-addons ejb3 session beans using default EJB3 > > interceptors. > > > > I wasn't aware that it was a requirement to have this file, so > > this is a bug, the intention is that this file is not > > required, but an option if you need to create an interceptor > > for some reason. And it really shouldn't be called > > metaloginterceptor, as it is really intercepting everything - > > this naming is related to our specific use case - we should > > change this however to a more generic name (maybe just > > interceptor). > > > > To tell you about our use case - we use interceptors to > > perform a parallell query on searchLogEntries (historic > > searches). We have a MetaDataLogEntry table with more than a > > billion of records, and the standard query produced by > > searchLogEntries is not able to handle this. The strategy is > > then to intercept the call, and duplicate it to run in > > multiple processes / servers on smaller dataset partitions, > > and collect the resultsets when sufficient amount of threads > > has returned. Then we also gain our desired speed (which is > > instant results). > > > > About table names - since the case problem is TopLink > > specific, could we rather solve this using an orm.xml for the > > unit tests? Instead of using the Table annotation? > > > > regards, > > > > Peter > > > > > > On 10/21/07, *Roberto MIlev* <rob...@gm... > > <mailto:rob...@gm...>> wrote: > > > > Hi Peter, > > > > You will find the changes you requested committed to SVN. > > A few notes: > > > > - I was only abe to test the shownColumns changes with > > dummy hardcoded > > data, beacuse I am getting errors due to missing resource - > > lmappletserver_metaloginterceptors. I am not able to find > > this resource > > in the project. Perhaps you can shed some light on this? > > - In the mean time I also resolved the issue with the junt > > tests > > failing. The thing was that the Toplink JPA always creates > > the tables > > based on the name of the Entity in uppercase. This was not > > visible until > > i install everything on Linux. the solution is to always > > specify the > > name attribute in the @Table annotation. > > > > Regards, > > Roberto > > > > Peter Salomonsen wrote: > > > Hi, > > > > > > I was just going to update my metadata table by adding a > > new metadata > > > (new column). I found that the searchdto has a > > showcolumns boolean > > > array, and this causes the requirement that the metadata > > cannot change > > > order or change in size. What happened in my case was that > > > listmetadata returned more columns than the showcolumns > > array, and > > > this caused an ArrayIndexOutOfBounds exception in > > > MetaDataSearchExecutor.java on the showcolumns array. > > > > > > I think this has to be solved differently. The > > showcolumns array > > > should rather be a list of metadata_ids rather than > > booleans - so that > > > it is not dependent on me not changing the metadata order > > or number of > > > metadatas. > > > > > > Can you fix this? > > > > > > regards, > > > > > > Peter > > > > > > > > > > > > > > -- > > Regards, > > Roberto Milev > > > > > > |
|
From: Roberto M. <rob...@gm...> - 2007-11-01 13:32:58
|
Hi Peter, About the folder structure, an alternative approach would be to organize it to reflect the actual MetaDataLogEntry id. for example: MetaDataLogEntry id = 1001 - would map to /1/0/0/1 MetaDataLogEntry id = 1002 - would map to /1/0/0/3 .. etc MetaDataLogEntry id = 1011 - would map to /1/0/1/1 in this manner every media file could be easily located within the data structure. I will start setting the infrastructure for the Phase One of the task. I will keep you posted of the progress. Regards, Roberto On 10/30/07, Peter Salomonsen <co...@pe...> wrote: > > Hi Roberto, > > I think we should split the project in two phases: > > Phase One: > > Media upload mechanisms and media file download (with correct mime types > etc) - but no media player. So the play button would go directly to the > multimediaservlet. > > Phase Two: > > Embedded Mediaplayer using e.g. FMJ. > > About the file sys: > > Splitting on date folders is good (to speed of access), but in case of > massive uploads, it might be better to split on MetaDataLogEntry id ranges. > Maybe the folder structure should reflect this - e.g. 1000 files per > folder (this could be customizable). > > I see the challenges regarding separating the project, and the challenge > regarding different data types we've also had with other data types. The > MetaDataLogEntry table should maybe have one field per datatype such as with > the numericvalue field - that we don't use today - but maybe we should. > > regards, > > Peter > > On 10/30/07, Roberto MIlev <rob...@gm...> wrote: > > > > Hi Peter, > > > > Please find attached my ideas on on implementing the new feature. My > > initial rough estimate for implementation is around 80 man hours. > > > > Please give me your thoughts. > > > > Regards, > > Roberto > > > > Peter Salomonsen wrote: > > > Hi Roberto - > > > > > > how is it going with the interceptor stuff - did you get to understand > > > how it works? > > > > > > I have a new project proposal for you. We'd like to add multimedia > > > capabilities to the MetaLogViewer - so that you can attach multimedia > > > clips to a MetaDataLogEntry. The idea will be to create new columns > > > that contains multimedia file names, and with a possibility to launch > > > a player to view /play. Embedded players would be very nice - however > > > supporting all the varieties of file formats may be difficult. Have to > > > consider whether to stick to one format ( e.g. ogg/vorbis/theora - and > > > > > use e.g. the flumotion cortado applet), or support multiple formats, > > > and test frameworks such as fmj (which is an open source impl of java > > > media framework). > > > > > > I don't think the media content itself should be stored in the > > > MetaDataLogEntry table, but the file name could be, and there should > > > be a file structure for easy retrieval. > > > > > > Get the idea? > > > > > > This makes the lmappletserver-addons.jar even more massive, and the > > > question is if we could create a new Multimedia-extensions module? > > > Should this even be a whole new separate project? > > > > > > Is this a project for you? In that case some sketches and models would > > > be nice, would you have a look at it, and think of a solution > > proposal? > > > > > > regards, > > > > > > Peter > > > > > > On 10/23/07, *Roberto Milev* <rob...@gm... > > > <mailto:rob...@gm...>> wrote: > > > > > > Hi Peter, > > > > > > I changed and commiter orm.xml. I will be looking at the > > > interceptor stuff next. I will keep you posted. > > > > > > Regards, > > > Roberto > > > > > > > > > On 10/23/07, * Peter Salomonsen* <co...@pe... > > > <mailto: co...@pe...>> wrote: > > > > > > Hi Robert, > > > > > > I've attached a metaloginterceptors property file example. > > > This is a feature me and Jon has been working on, and the > > > purpose is to be able to intercept calls within the > > > lmappletserver-addons ejb3 session beans using default EJB3 > > > interceptors. > > > > > > I wasn't aware that it was a requirement to have this file, so > > > > > this is a bug, the intention is that this file is not > > > required, but an option if you need to create an interceptor > > > for some reason. And it really shouldn't be called > > > metaloginterceptor, as it is really intercepting everything - > > > this naming is related to our specific use case - we should > > > change this however to a more generic name (maybe just > > > interceptor). > > > > > > To tell you about our use case - we use interceptors to > > > perform a parallell query on searchLogEntries (historic > > > searches). We have a MetaDataLogEntry table with more than a > > > billion of records, and the standard query produced by > > > searchLogEntries is not able to handle this. The strategy is > > > then to intercept the call, and duplicate it to run in > > > multiple processes / servers on smaller dataset partitions, > > > and collect the resultsets when sufficient amount of threads > > > has returned. Then we also gain our desired speed (which is > > > instant results). > > > > > > About table names - since the case problem is TopLink > > > specific, could we rather solve this using an orm.xml for the > > > unit tests? Instead of using the Table annotation? > > > > > > regards, > > > > > > Peter > > > > > > > > > On 10/21/07, *Roberto MIlev* <rob...@gm... > > > <mailto: rob...@gm...>> wrote: > > > > > > Hi Peter, > > > > > > You will find the changes you requested committed to SVN. > > > A few notes: > > > > > > - I was only abe to test the shownColumns changes with > > > dummy hardcoded > > > data, beacuse I am getting errors due to missing resource > > - > > > lmappletserver_metaloginterceptors. I am not able to find > > > this resource > > > in the project. Perhaps you can shed some light on this? > > > - In the mean time I also resolved the issue with the junt > > > tests > > > failing. The thing was that the Toplink JPA always creates > > > > > the tables > > > based on the name of the Entity in uppercase. This was not > > > visible until > > > i install everything on Linux. the solution is to always > > > specify the > > > name attribute in the @Table annotation. > > > > > > Regards, > > > Roberto > > > > > > Peter Salomonsen wrote: > > > > Hi, > > > > > > > > I was just going to update my metadata table by adding a > > > new metadata > > > > (new column). I found that the searchdto has a > > > showcolumns boolean > > > > array, and this causes the requirement that the metadata > > > cannot change > > > > order or change in size. What happened in my case was > > that > > > > listmetadata returned more columns than the showcolumns > > > array, and > > > > this caused an ArrayIndexOutOfBounds exception in > > > > MetaDataSearchExecutor.java on the showcolumns array. > > > > > > > > I think this has to be solved differently. The > > > showcolumns array > > > > should rather be a list of metadata_ids rather than > > > booleans - so that > > > > it is not dependent on me not changing the metadata > > order > > > or number of > > > > metadatas. > > > > > > > > Can you fix this? > > > > > > > > regards, > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Roberto Milev > > > > > > > > > > > -- Regards, Roberto Milev |
|
From: Peter S. <co...@pe...> - 2007-11-02 07:33:31
|
Hi Roberto, I'm not a file system expert - so I cannot tell whats the most optimal regarding how many files to put in a folder - but your suggestion sounds interesting. This means that we also can keep actual media file names since each log entry has its own folder. I'd also like a comment from Jon on this. cheers, Peter On 11/1/07, Roberto Milev <rob...@gm...> wrote: > > Hi Peter, > > About the folder structure, an alternative approach would be to organize > it to reflect the actual MetaDataLogEntry id. for example: > > MetaDataLogEntry id = 1001 - would map to /1/0/0/1 > MetaDataLogEntry id = 1002 - would map to /1/0/0/3 > .. etc > MetaDataLogEntry id = 1011 - would map to /1/0/1/1 > > in this manner every media file could be easily located within the data > structure. > > I will start setting the infrastructure for the Phase One of the task. I > will keep you posted of the progress. > > Regards, > Roberto > > > On 10/30/07, Peter Salomonsen <co...@pe...> wrote: > > > > Hi Roberto, > > > > I think we should split the project in two phases: > > > > Phase One: > > > > Media upload mechanisms and media file download (with correct mime types > > etc) - but no media player. So the play button would go directly to the > > multimediaservlet. > > > > Phase Two: > > > > Embedded Mediaplayer using e.g. FMJ. > > > > About the file sys: > > > > Splitting on date folders is good (to speed of access), but in case of > > massive uploads, it might be better to split on MetaDataLogEntry id ranges. > > Maybe the folder structure should reflect this - e.g. 1000 files per > > folder (this could be customizable). > > > > I see the challenges regarding separating the project, and the challenge > > regarding different data types we've also had with other data types. The > > MetaDataLogEntry table should maybe have one field per datatype such as with > > the numericvalue field - that we don't use today - but maybe we should. > > > > regards, > > > > Peter > > > > On 10/30/07, Roberto MIlev < rob...@gm...> wrote: > > > > > > Hi Peter, > > > > > > Please find attached my ideas on on implementing the new feature. My > > > initial rough estimate for implementation is around 80 man hours. > > > > > > Please give me your thoughts. > > > > > > Regards, > > > Roberto > > > > > > Peter Salomonsen wrote: > > > > Hi Roberto - > > > > > > > > how is it going with the interceptor stuff - did you get to > > > understand > > > > how it works? > > > > > > > > I have a new project proposal for you. We'd like to add multimedia > > > > capabilities to the MetaLogViewer - so that you can attach > > > multimedia > > > > clips to a MetaDataLogEntry. The idea will be to create new columns > > > > that contains multimedia file names, and with a possibility to > > > launch > > > > a player to view /play. Embedded players would be very nice - > > > however > > > > supporting all the varieties of file formats may be difficult. Have > > > to > > > > consider whether to stick to one format ( e.g. ogg/vorbis/theora - > > > and > > > > use e.g. the flumotion cortado applet), or support multiple formats, > > > > and test frameworks such as fmj (which is an open source impl of > > > java > > > > media framework). > > > > > > > > I don't think the media content itself should be stored in the > > > > MetaDataLogEntry table, but the file name could be, and there should > > > > be a file structure for easy retrieval. > > > > > > > > Get the idea? > > > > > > > > This makes the lmappletserver-addons.jar even more massive, and the > > > > > > > question is if we could create a new Multimedia-extensions module? > > > > Should this even be a whole new separate project? > > > > > > > > Is this a project for you? In that case some sketches and models > > > would > > > > be nice, would you have a look at it, and think of a solution > > > proposal? > > > > > > > > regards, > > > > > > > > Peter > > > > > > > > On 10/23/07, *Roberto Milev* < rob...@gm... > > > > <mailto:rob...@gm...>> wrote: > > > > > > > > Hi Peter, > > > > > > > > I changed and commiter orm.xml. I will be looking at the > > > > interceptor stuff next. I will keep you posted. > > > > > > > > Regards, > > > > Roberto > > > > > > > > > > > > On 10/23/07, * Peter Salomonsen* < co...@pe... > > > > <mailto: co...@pe...>> wrote: > > > > > > > > Hi Robert, > > > > > > > > I've attached a metaloginterceptors property file example. > > > > This is a feature me and Jon has been working on, and the > > > > purpose is to be able to intercept calls within the > > > > lmappletserver-addons ejb3 session beans using default EJB3 > > > > interceptors. > > > > > > > > I wasn't aware that it was a requirement to have this file, > > > so > > > > this is a bug, the intention is that this file is not > > > > required, but an option if you need to create an interceptor > > > > for some reason. And it really shouldn't be called > > > > metaloginterceptor, as it is really intercepting everything > > > - > > > > this naming is related to our specific use case - we should > > > > change this however to a more generic name (maybe just > > > > interceptor). > > > > > > > > To tell you about our use case - we use interceptors to > > > > perform a parallell query on searchLogEntries (historic > > > > searches). We have a MetaDataLogEntry table with more than a > > > > > > > billion of records, and the standard query produced by > > > > searchLogEntries is not able to handle this. The strategy is > > > > then to intercept the call, and duplicate it to run in > > > > multiple processes / servers on smaller dataset partitions, > > > > and collect the resultsets when sufficient amount of threads > > > > has returned. Then we also gain our desired speed (which is > > > > instant results). > > > > > > > > About table names - since the case problem is TopLink > > > > specific, could we rather solve this using an orm.xml for > > > the > > > > unit tests? Instead of using the Table annotation? > > > > > > > > regards, > > > > > > > > Peter > > > > > > > > > > > > On 10/21/07, *Roberto MIlev* < rob...@gm... > > > > <mailto: rob...@gm...>> wrote: > > > > > > > > Hi Peter, > > > > > > > > You will find the changes you requested committed to > > > SVN. > > > > A few notes: > > > > > > > > - I was only abe to test the shownColumns changes with > > > > dummy hardcoded > > > > data, beacuse I am getting errors due to missing > > > resource - > > > > lmappletserver_metaloginterceptors. I am not able to > > > find > > > > this resource > > > > in the project. Perhaps you can shed some light on this? > > > > - In the mean time I also resolved the issue with the > > > junt > > > > tests > > > > failing. The thing was that the Toplink JPA always > > > creates > > > > the tables > > > > based on the name of the Entity in uppercase. This was > > > not > > > > visible until > > > > i install everything on Linux. the solution is to always > > > > > > > specify the > > > > name attribute in the @Table annotation. > > > > > > > > Regards, > > > > Roberto > > > > > > > > Peter Salomonsen wrote: > > > > > Hi, > > > > > > > > > > I was just going to update my metadata table by adding > > > a > > > > new metadata > > > > > (new column). I found that the searchdto has a > > > > showcolumns boolean > > > > > array, and this causes the requirement that the > > > metadata > > > > cannot change > > > > > order or change in size. What happened in my case was > > > that > > > > > listmetadata returned more columns than the > > > showcolumns > > > > array, and > > > > > this caused an ArrayIndexOutOfBounds exception in > > > > > MetaDataSearchExecutor.java on the showcolumns array. > > > > > > > > > > I think this has to be solved differently. The > > > > showcolumns array > > > > > should rather be a list of metadata_ids rather than > > > > booleans - so that > > > > > it is not dependent on me not changing the metadata > > > order > > > > or number of > > > > > metadatas. > > > > > > > > > > Can you fix this? > > > > > > > > > > regards, > > > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Roberto Milev > > > > > > > > > > > > > > > > > > > -- > Regards, > Roberto Milev |
|
From: Roberto M. <rob...@gm...> - 2007-11-07 23:19:35
|
Hi Peter,
Here is a status on the progress so far:
1. What has been done so far.
* A new record in the metadata database teble is added with "mmx"
dataType value:
insert into
`metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`)
values (1015,'Multimedia','',9,'mmx','
com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27
11:42:30','2006-02-27 11:42:30','','\0');
* The metadatalogentry gui table has the option when a column has a
"mmx" dataType to show button instead of label cell. That button
leads to downloading servlet (for now).
* Streaming(download) and uploading servlets are created in new
project named lmappletserver_multimedia. The corresponding changes
are made in lmappletserver_build project to include the multimedia
project.
2. What has not been done so far:
* The uploading servlet does not place the files in folders as we
discussed.
* There is no button or link to lead you to the upload servlet.
* Configuration parameters are still hardcoded in MediaType class
(they will be in property file at the end as report_conf.properties)
3. Questions:
* We didn't finally agree on how to make the folder structure,
please tell me what approach to use?
* Is it good to use the dataType metadata field to represent
multimedia column, or we need to add another column?
* On the upload servlet now there is a field that gets entryId, for
that entryId the file is added in the database, is this OK?
Please give me your comments.
Regards,
Roberto
Peter Salomonsen wrote:
> Hi Roberto,
>
> I'm not a file system expert - so I cannot tell whats the most optimal
> regarding how many files to put in a folder - but your suggestion
> sounds interesting. This means that we also can keep actual media file
> names since each log entry has its own folder.
>
> I'd also like a comment from Jon on this.
>
> cheers,
>
> Peter
>
> On 11/1/07, *Roberto Milev* < rob...@gm...
> <mailto:rob...@gm...>> wrote:
>
> Hi Peter,
>
> About the folder structure, an alternative approach would be to
> organize it to reflect the actual MetaDataLogEntry id. for example:
>
> MetaDataLogEntry id = 1001 - would map to /1/0/0/1
> MetaDataLogEntry id = 1002 - would map to /1/0/0/3
> .. etc
> MetaDataLogEntry id = 1011 - would map to /1/0/1/1
>
> in this manner every media file could be easily located within the
> data structure.
>
> I will start setting the infrastructure for the Phase One of the
> task. I will keep you posted of the progress.
>
> Regards,
> Roberto
>
>
>
> On 10/30/07, *Peter Salomonsen* < co...@pe...
> <mailto:co...@pe...>> wrote:
>
> Hi Roberto,
>
> I think we should split the project in two phases:
>
> Phase One:
>
> Media upload mechanisms and media file download (with correct
> mime types etc) - but no media player. So the play button
> would go directly to the multimediaservlet.
>
> Phase Two:
>
> Embedded Mediaplayer using e.g. FMJ.
>
> About the file sys:
>
> Splitting on date folders is good (to speed of access), but in
> case of massive uploads, it might be better to split on
> MetaDataLogEntry id ranges. Maybe the folder structure should
> reflect this - e.g. 1000 files per folder (this could be
> customizable).
>
> I see the challenges regarding separating the project, and the
> challenge regarding different data types we've also had with
> other data types. The MetaDataLogEntry table should maybe have
> one field per datatype such as with the numericvalue field -
> that we don't use today - but maybe we should.
>
> regards,
>
> Peter
>
>
> On 10/30/07, *Roberto MIlev* < rob...@gm...
> <mailto:rob...@gm...>> wrote:
>
> Hi Peter,
>
> Please find attached my ideas on on implementing the new
> feature. My
> initial rough estimate for implementation is around 80 man
> hours.
>
> Please give me your thoughts.
>
> Regards,
> Roberto
>
> Peter Salomonsen wrote:
> > Hi Roberto -
> >
> > how is it going with the interceptor stuff - did you get
> to understand
> > how it works?
> >
> > I have a new project proposal for you. We'd like to add
> multimedia
> > capabilities to the MetaLogViewer - so that you can
> attach multimedia
> > clips to a MetaDataLogEntry. The idea will be to create
> new columns
> > that contains multimedia file names, and with a
> possibility to launch
> > a player to view /play. Embedded players would be very
> nice - however
> > supporting all the varieties of file formats may be
> difficult. Have to
> > consider whether to stick to one format ( e.g.
> ogg/vorbis/theora - and
> > use e.g. the flumotion cortado applet), or support
> multiple formats,
> > and test frameworks such as fmj (which is an open source
> impl of java
> > media framework).
> >
> > I don't think the media content itself should be stored
> in the
> > MetaDataLogEntry table, but the file name could be, and
> there should
> > be a file structure for easy retrieval.
> >
> > Get the idea?
> >
> > This makes the lmappletserver-addons.jar even more
> massive, and the
> > question is if we could create a new
> Multimedia-extensions module?
> > Should this even be a whole new separate project?
> >
> > Is this a project for you? In that case some sketches and
> models would
> > be nice, would you have a look at it, and think of a
> solution proposal?
> >
> > regards,
> >
> > Peter
> >
> > On 10/23/07, *Roberto Milev* < rob...@gm...
> <mailto:rob...@gm...>
> > <mailto:rob...@gm...
> <mailto:rob...@gm...>>> wrote:
> >
> > Hi Peter,
> >
> > I changed and commiter orm.xml. I will be looking at the
> > interceptor stuff next. I will keep you posted.
> >
> > Regards,
> > Roberto
> >
> >
> > On 10/23/07, * Peter Salomonsen* <
> co...@pe...
> <mailto:co...@pe...>
> > <mailto: co...@pe...
> <mailto:co...@pe...>>> wrote:
> >
> > Hi Robert,
> >
> > I've attached a metaloginterceptors property file
> example.
> > This is a feature me and Jon has been working on,
> and the
> > purpose is to be able to intercept calls within the
> > lmappletserver-addons ejb3 session beans using
> default EJB3
> > interceptors.
> >
> > I wasn't aware that it was a requirement to have
> this file, so
> > this is a bug, the intention is that this file is not
> > required, but an option if you need to create an
> interceptor
> > for some reason. And it really shouldn't be called
> > metaloginterceptor, as it is really intercepting
> everything -
> > this naming is related to our specific use case -
> we should
> > change this however to a more generic name (maybe
> just
> > interceptor).
> >
> > To tell you about our use case - we use
> interceptors to
> > perform a parallell query on searchLogEntries
> (historic
> > searches). We have a MetaDataLogEntry table with
> more than a
> > billion of records, and the standard query
> produced by
> > searchLogEntries is not able to handle this. The
> strategy is
> > then to intercept the call, and duplicate it to
> run in
> > multiple processes / servers on smaller dataset
> partitions,
> > and collect the resultsets when sufficient amount
> of threads
> > has returned. Then we also gain our desired speed
> (which is
> > instant results).
> >
> > About table names - since the case problem is TopLink
> > specific, could we rather solve this using an
> orm.xml for the
> > unit tests? Instead of using the Table annotation?
> >
> > regards,
> >
> > Peter
> >
> >
> > On 10/21/07, *Roberto MIlev* <
> rob...@gm... <mailto:rob...@gm...>
> > <mailto: rob...@gm...
> <mailto:rob...@gm...>>> wrote:
> >
> > Hi Peter,
> >
> > You will find the changes you requested
> committed to SVN.
> > A few notes:
> >
> > - I was only abe to test the shownColumns
> changes with
> > dummy hardcoded
> > data, beacuse I am getting errors due to
> missing resource -
> > lmappletserver_metaloginterceptors. I am not
> able to find
> > this resource
> > in the project. Perhaps you can shed some
> light on this?
> > - In the mean time I also resolved the issue
> with the junt
> > tests
> > failing. The thing was that the Toplink JPA
> always creates
> > the tables
> > based on the name of the Entity in uppercase.
> This was not
> > visible until
> > i install everything on Linux. the solution
> is to always
> > specify the
> > name attribute in the @Table annotation.
> >
> > Regards,
> > Roberto
> >
> > Peter Salomonsen wrote:
> > > Hi,
> > >
> > > I was just going to update my metadata
> table by adding a
> > new metadata
> > > (new column). I found that the searchdto has a
> > showcolumns boolean
> > > array, and this causes the requirement that
> the metadata
> > cannot change
> > > order or change in size. What happened in
> my case was that
> > > listmetadata returned more columns than the
> showcolumns
> > array, and
> > > this caused an ArrayIndexOutOfBounds
> exception in
> > > MetaDataSearchExecutor.java on the
> showcolumns array.
> > >
> > > I think this has to be solved differently. The
> > showcolumns array
> > > should rather be a list of metadata_ids
> rather than
> > booleans - so that
> > > it is not dependent on me not changing the
> metadata order
> > or number of
> > > metadatas.
> > >
> > > Can you fix this?
> > >
> > > regards,
> > >
> > > Peter
> >
> >
> >
> >
> >
> >
> > --
> > Regards,
> > Roberto Milev
> >
> >
>
>
>
>
>
> --
> Regards,
> Roberto Milev
>
>
|
|
From: Peter S. <co...@pe...> - 2007-11-08 10:23:43
|
See comments below. On Nov 8, 2007 12:19 AM, Roberto MIlev <rob...@gm...> wrote: > Hi Peter, > > Here is a status on the progress so far: > > 1. What has been done so far. > > > * A new record in the metadata database teble is added with "mmx" > dataType value: > > insert into > > `metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`) > values (1015,'Multimedia','',9,'mmx',' > com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27 > 11:42:30','2006-02-27 11:42:30','','\0'); This is good. > > * The metadatalogentry gui table has the option when a column has a > "mmx" dataType to show button instead of label cell. That button > leads to downloading servlet (for now). Sounds good. > > * Streaming(download) and uploading servlets are created in new > project named lmappletserver_multimedia. The corresponding changes > are made in lmappletserver_build project to include the multimedia > project. > ok > > 2. What has not been done so far: > > * The uploading servlet does not place the files in folders as we > discussed. > * There is no button or link to lead you to the upload servlet. The idea is that one should use a e.g. mobile phone application to do this. So you have to have this in mind, that the upload interface should be reachable by third party apps. As a test case you can make a simple web form. > * Configuration parameters are still hardcoded in MediaType class > (they will be in property file at the end as report_conf.properties) > > 3. Questions: > > * We didn't finally agree on how to make the folder structure, > please tell me what approach to use? I think your last suggestion was good. Just make it to log_id rather than entry_id. So that if there are more than one multimedia file per log_entry they will be in the same folder. does that sound ok? > > * Is it good to use the dataType metadata field to represent > multimedia column, or we need to add another column? Yes I think thats the best way. > > * On the upload servlet now there is a field that gets entryId, for > that entryId the file is added in the database, is this OK? use log_id rather than entry_id - I probably think is better? regards, Peter |
|
From: Roberto M. <rob...@gm...> - 2007-11-09 00:08:17
|
Hi Peter, I agree with your comments. One small thing though, when I sad entry_id I meant object_id. I have to get the object_id form the upload page because the addLogEntry needs it. And by using that I can easy create new folders or add files to existing ones. That is why I thought it is a better idea to use the object_id as folder coordinate. However, as things are now I add every file a random integer number at the beginning of the file when writing it to the FS and DB, so there is no problem with duplicate naming. Sorry once again for the miss leading and please tell me if my conclusion is fine. Regards, Roberto Peter Salomonsen wrote: > See comments below. > > On Nov 8, 2007 12:19 AM, Roberto MIlev <rob...@gm... > <mailto:rob...@gm...>> wrote: > > Hi Peter, > > Here is a status on the progress so far: > > 1. What has been done so far. > > > * A new record in the metadata database teble is added with "mmx" > dataType value: > > insert into > `metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`) > values (1015,'Multimedia','',9,'mmx',' > com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27 > 11:42:30','2006-02-27 11:42:30','','\0'); > > > This is good. > > > > * The metadatalogentry gui table has the option when a column has a > "mmx" dataType to show button instead of label cell. That button > leads to downloading servlet (for now). > > > Sounds good. > > > > * Streaming(download) and uploading servlets are created in new > project named lmappletserver_multimedia. The corresponding > changes > are made in lmappletserver_build project to include the > multimedia > project. > > ok > > > > 2. What has not been done so far: > > * The uploading servlet does not place the files in folders as we > discussed. > * There is no button or link to lead you to the upload servlet. > > The idea is that one should use a e.g. mobile phone application to do > this. So you have to have this in mind, that the upload interface > should be reachable by third party apps. As a test case you can make a > simple web form. > > > * Configuration parameters are still hardcoded in MediaType class > (they will be in property file at the end as > report_conf.properties) > > 3. Questions: > > * We didn't finally agree on how to make the folder structure, > please tell me what approach to use? > > > I think your last suggestion was good. Just make it to log_id rather > than entry_id. So that if there are more than one multimedia file per > log_entry they will be in the same folder. does that sound ok? > > > > * Is it good to use the dataType metadata field to represent > multimedia column, or we need to add another column? > > > Yes I think thats the best way. > > > > * On the upload servlet now there is a field that gets entryId, > for > that entryId the file is added in the database, is this OK? > > > use log_id rather than entry_id - I probably think is better? > > > regards, > > Peter |
|
From: Peter S. <co...@pe...> - 2007-11-09 08:28:54
|
OK - so you mean one folder per object (e.g. Vehicle entity) - I see.. Only drawback I see with that - is that if we say that the images /movies come from a surveillance camera - and the snapshots are very frequent - it can be maaany files for one camera, and then also many files for that object. In that sense log_id would probably be better? One use case might even be that you take photos, extract metadata about the photo (time,place,colors,if anything is moving etc.) and enable search on these metadatas as well as displaying the images. You might imagine that it can be very many images for one object. regards, Peter On Nov 9, 2007 1:08 AM, Roberto MIlev <rob...@gm...> wrote: > Hi Peter, > > I agree with your comments. One small thing though, when I sad entry_id > I meant object_id. I have to get the object_id form the upload page > because the addLogEntry needs it. And by using that I can easy create > new folders or add files to existing ones. That is why I thought it is a > better idea to use the object_id as folder coordinate. However, as > things are now I add every file a random integer number at the beginning > of the file when writing it to the FS and DB, so there is no problem > with duplicate naming. Sorry once again for the miss leading and please > tell me if my conclusion is fine. > > Regards, > Roberto > > Peter Salomonsen wrote: > > See comments below. > > > > On Nov 8, 2007 12:19 AM, Roberto MIlev <rob...@gm... > > <mailto:rob...@gm...>> wrote: > > > > Hi Peter, > > > > Here is a status on the progress so far: > > > > 1. What has been done so far. > > > > > > * A new record in the metadata database teble is added with "mmx" > > dataType value: > > > > insert into > > > `metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`) > > values (1015,'Multimedia','',9,'mmx',' > > com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27 > > 11:42:30','2006-02-27 11:42:30','','\0'); > > > > > > This is good. > > > > > > > > * The metadatalogentry gui table has the option when a column has > a > > "mmx" dataType to show button instead of label cell. That > button > > leads to downloading servlet (for now). > > > > > > Sounds good. > > > > > > > > * Streaming(download) and uploading servlets are created in new > > project named lmappletserver_multimedia. The corresponding > > changes > > are made in lmappletserver_build project to include the > > multimedia > > project. > > > > ok > > > > > > > > 2. What has not been done so far: > > > > * The uploading servlet does not place the files in folders as we > > discussed. > > * There is no button or link to lead you to the upload servlet. > > > > The idea is that one should use a e.g. mobile phone application to do > > this. So you have to have this in mind, that the upload interface > > should be reachable by third party apps. As a test case you can make a > > simple web form. > > > > > > * Configuration parameters are still hardcoded in MediaType class > > (they will be in property file at the end as > > report_conf.properties) > > > > 3. Questions: > > > > * We didn't finally agree on how to make the folder structure, > > please tell me what approach to use? > > > > > > I think your last suggestion was good. Just make it to log_id rather > > than entry_id. So that if there are more than one multimedia file per > > log_entry they will be in the same folder. does that sound ok? > > > > > > > > * Is it good to use the dataType metadata field to represent > > multimedia column, or we need to add another column? > > > > > > Yes I think thats the best way. > > > > > > > > * On the upload servlet now there is a field that gets entryId, > > for > > that entryId the file is added in the database, is this OK? > > > > > > use log_id rather than entry_id - I probably think is better? > > > > > > regards, > > > > Peter > |
|
From: Roberto M. <rob...@gm...> - 2007-11-13 03:53:45
|
Hi Peter, All changes regarding phase-one are implemented and committed. You can have a look and let me know what you think. Next I will be continuing my work on the applet. Regards, Roberto Peter Salomonsen wrote: > OK - so you mean one folder per object (e.g. Vehicle entity) - I > see.. Only drawback I see with that - is that if we say that the > images /movies come from a surveillance camera - and the snapshots are > very frequent - it can be maaany files for one camera, and then also > many files for that object. In that sense log_id would probably be > better? > > One use case might even be that you take photos, extract metadata > about the photo (time,place,colors,if anything is moving etc.) and > enable search on these metadatas as well as displaying the images. You > might imagine that it can be very many images for one object. > > regards, > > Peter > > On Nov 9, 2007 1:08 AM, Roberto MIlev <rob...@gm... > <mailto:rob...@gm...>> wrote: > > Hi Peter, > > I agree with your comments. One small thing though, when I sad > entry_id > I meant object_id. I have to get the object_id form the upload page > because the addLogEntry needs it. And by using that I can easy create > new folders or add files to existing ones. That is why I thought > it is a > better idea to use the object_id as folder coordinate. However, as > things are now I add every file a random integer number at the > beginning > of the file when writing it to the FS and DB, so there is no problem > with duplicate naming. Sorry once again for the miss leading and > please > tell me if my conclusion is fine. > > Regards, > Roberto > > Peter Salomonsen wrote: > > See comments below. > > > > On Nov 8, 2007 12:19 AM, Roberto MIlev <rob...@gm... > <mailto:rob...@gm...> > > <mailto:rob...@gm... > <mailto:rob...@gm...>>> wrote: > > > > Hi Peter, > > > > Here is a status on the progress so far: > > > > 1. What has been done so far. > > > > > > * A new record in the metadata database teble is added > with "mmx" > > dataType value: > > > > insert into > > > `metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`) > > values (1015,'Multimedia','',9,'mmx',' > > > com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27 > > 11:42:30','2006-02-27 11:42:30','','\0'); > > > > > > This is good. > > > > > > > > * The metadatalogentry gui table has the option when a > column has a > > "mmx" dataType to show button instead of label cell. > That button > > leads to downloading servlet (for now). > > > > > > Sounds good. > > > > > > > > * Streaming(download) and uploading servlets are created > in new > > project named lmappletserver_multimedia. The corresponding > > changes > > are made in lmappletserver_build project to include the > > multimedia > > project. > > > > ok > > > > > > > > 2. What has not been done so far: > > > > * The uploading servlet does not place the files in > folders as we > > discussed. > > * There is no button or link to lead you to the upload > servlet. > > > > The idea is that one should use a e.g. mobile phone application > to do > > this. So you have to have this in mind, that the upload interface > > should be reachable by third party apps. As a test case you can > make a > > simple web form. > > > > > > * Configuration parameters are still hardcoded in > MediaType class > > (they will be in property file at the end as > > report_conf.properties) > > > > 3. Questions: > > > > * We didn't finally agree on how to make the folder > structure, > > please tell me what approach to use? > > > > > > I think your last suggestion was good. Just make it to log_id > rather > > than entry_id. So that if there are more than one multimedia > file per > > log_entry they will be in the same folder. does that sound ok? > > > > > > > > * Is it good to use the dataType metadata field to represent > > multimedia column, or we need to add another column? > > > > > > Yes I think thats the best way. > > > > > > > > * On the upload servlet now there is a field that gets > entryId, > > for > > that entryId the file is added in the database, is this OK? > > > > > > use log_id rather than entry_id - I probably think is better? > > > > > > regards, > > > > Peter > > |
|
From: Roberto M. <rob...@gm...> - 2007-11-16 01:45:02
|
Hi Peter, I finished both parts of the multimedia addon. I had a few problems during the development and testing. I know that I proposed to use FMJ, but it appears to have many disadvantages. For example the documentation says that FMJ jars dont need JMF, in fact they do, they need a lot of the JMF drivers and native code. And when you install it the lib path of FMJ must be in front of the JMF path in the classpath at the client. I had an enormous number of problems until I finally decided to try pure JMF. When I did, it appeared that the only downside of not using FMJ is the support for GG format. If you think that this is a "must have" format, than we will seek some other way, using FMJ brings a lot of trouble. Another thing is that the JFM must be installed at client side. We can not use the whole JARServlet mechanism. These jars use native code and dll files, which is problem for the JARServlet. I tried to fix this but I kept getting the same error Problem loading JMF registry - Unable to initialize ffmpeg libraries. If the client has a JMF instaled, than everything is OK. I committed changes as they are described above with JMF. I used a SimplePlayerApplet and it has only play and pause buttons for now. Please tell me how to proceed? Peter Salomonsen wrote: > OK - so you mean one folder per object (e.g. Vehicle entity) - I > see.. Only drawback I see with that - is that if we say that the > images /movies come from a surveillance camera - and the snapshots are > very frequent - it can be maaany files for one camera, and then also > many files for that object. In that sense log_id would probably be > better? > > One use case might even be that you take photos, extract metadata > about the photo (time,place,colors,if anything is moving etc.) and > enable search on these metadatas as well as displaying the images. You > might imagine that it can be very many images for one object. > > regards, > > Peter > > On Nov 9, 2007 1:08 AM, Roberto MIlev <rob...@gm... > <mailto:rob...@gm...>> wrote: > > Hi Peter, > > I agree with your comments. One small thing though, when I sad > entry_id > I meant object_id. I have to get the object_id form the upload page > because the addLogEntry needs it. And by using that I can easy create > new folders or add files to existing ones. That is why I thought > it is a > better idea to use the object_id as folder coordinate. However, as > things are now I add every file a random integer number at the > beginning > of the file when writing it to the FS and DB, so there is no problem > with duplicate naming. Sorry once again for the miss leading and > please > tell me if my conclusion is fine. > > Regards, > Roberto > > Peter Salomonsen wrote: > > See comments below. > > > > On Nov 8, 2007 12:19 AM, Roberto MIlev <rob...@gm... > <mailto:rob...@gm...> > > <mailto:rob...@gm... > <mailto:rob...@gm...>>> wrote: > > > > Hi Peter, > > > > Here is a status on the progress so far: > > > > 1. What has been done so far. > > > > > > * A new record in the metadata database teble is added > with "mmx" > > dataType value: > > > > insert into > > > `metadata`(`meta_id`,`name`,`context`,`priority`,`dataType`,`object_class_name`,`created`,`updated`,`searchProperty`,`numericValue`) > > values (1015,'Multimedia','',9,'mmx',' > > > com.lightminds.appletserver.examples.metalog.Vehicle','2006-02-27 > > 11:42:30','2006-02-27 11:42:30','','\0'); > > > > > > This is good. > > > > > > > > * The metadatalogentry gui table has the option when a > column has a > > "mmx" dataType to show button instead of label cell. > That button > > leads to downloading servlet (for now). > > > > > > Sounds good. > > > > > > > > * Streaming(download) and uploading servlets are created > in new > > project named lmappletserver_multimedia. The corresponding > > changes > > are made in lmappletserver_build project to include the > > multimedia > > project. > > > > ok > > > > > > > > 2. What has not been done so far: > > > > * The uploading servlet does not place the files in > folders as we > > discussed. > > * There is no button or link to lead you to the upload > servlet. > > > > The idea is that one should use a e.g. mobile phone application > to do > > this. So you have to have this in mind, that the upload interface > > should be reachable by third party apps. As a test case you can > make a > > simple web form. > > > > > > * Configuration parameters are still hardcoded in > MediaType class > > (they will be in property file at the end as > > report_conf.properties) > > > > 3. Questions: > > > > * We didn't finally agree on how to make the folder > structure, > > please tell me what approach to use? > > > > > > I think your last suggestion was good. Just make it to log_id > rather > > than entry_id. So that if there are more than one multimedia > file per > > log_entry they will be in the same folder. does that sound ok? > > > > > > > > * Is it good to use the dataType metadata field to represent > > multimedia column, or we need to add another column? > > > > > > Yes I think thats the best way. > > > > > > > > * On the upload servlet now there is a field that gets > entryId, > > for > > that entryId the file is added in the database, is this OK? > > > > > > use log_id rather than entry_id - I probably think is better? > > > > > > regards, > > > > Peter > > |