I am setting up a htpc, so this is something that I have put alot of thought into. Your idea is similar enough to mine that I have set your project page as my home page (as opposed to the blank page that I have had for the past year)
It seems to me that a successful implementation of such an environment may manifest itself in such a manner.
Your webpage says that you are not trying to re-invent the wheel, however, I think that the ability to
re-invent the wheel is a necessity, albeit, a
possibility that does not need to be exploited.
I imagine an environment which is made up of four categories of plugins. It should be possible to run any of these plugins simultaneously, and even run it's output into the input of another plugin in the same category.
Input - any variety of inputs... tv tuner, disk reader, internet transport, analog audio, keyboard, oystick, nintendo power glove... all it does is take the input, and pass it to the decode plugins, encapsulated in
a standardized transport for all of them.
Decode - takes input from the input plugins, and formats it so that it can be processed... a standard way of presenting video to the processors, a standard way of presenting audio to the processors,
a standard way of presenting user input...
Processing Plugins that can work independant of the data source, deinterlacers, dsp filters, databases, anything you can think of that can be done to data in realtime, should be able to be written as a processing plugin.
Output
The data gets passed to the output filters in the same format thay hit the processing filters. And is then formatted by these plugins for any imaginable output device. Direct sound, OSS, OpenGl, Directx, DirectFB, Writing directly to the video memory in 320x200 8 bit mode, blinking lights on your keyboard, and a disk writer, which might get it's data from another output codec, which would encode it into divx,mp3,wav,etc.
A sample of how I envision this may work in one situation would be..
Input from a tv tuner card.
The V4L Tv tuner input plugin grabs the data from V4L and wraps it up to send to the V4L Decoder. The V4L decoder takes the data and splits it into a seperate standard video and audio streams. The video stream gets run through a deinterlacer, a
noise filter, and a timeshifter. The audio stream gets sent to a equalizer, and 3d effect dsp.
Both the video stream is sent to the DirectFB output, the audio stream is sent to OSS. The video and audio streams are also sent to the Mpeg4 encoder plugin, which outputs to the disk writer plugin and the mpeg4 server plugin.
The user watches tv while it is recorded, and streamed out over the internet. Switching formats is easy, adding any kind of input you want is easy.
The interface would also be a processing plugin. Perhaps you could hit a button on a remote, which gets caught by a serial reciever, which gets sent to the serial input plugin, which goes to the IR decoder plugin, which goes to the Flashy interface processing plugin, which querys the database plugin (which gets the database from maybe the disk reader, or the http input plugin), and returns a file location which goes to the disk reader input plugin, which gets the next mp3, and sends it to the mp3 decoder plugin, which goes to the 3d effect dsp processing plugin, then to the OSS output plugin. The Flashy interface processing plugin gets sent to the Direct FB output plugin.
Overlaying video and audio streams could be configured in their respective output plugins.
Sorry, that was long.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In fact it is very, very similar to what I have in mind. You are going a bit deeper though that I first intended.
To begin with I would separate the "interface" (intended in a broad sense, not just the GUI, and including low level user interfaces, multimedia DB, file management utilities, scheduler, info grabbers, epg and other typical HTPC utilities), from the media players themself.
Both the "interface" and the players can probably be implemented in a plug-in system similar to the one you descibe but the separation is mostly justifed by the type of I/O stream between components. One thing is to send a synchronized A/V stream between two components and another is to send a simple xml string.
For a full blown HTPC the "interface" is quite complex as it includes several components itself (components that are sometimes "fixed" like the multimedia DB, and sometime "swappable" like the info grabbers or low level UI) and it should be as independent as possible from the players actually used to process the A/V stream.
I was focusing mostly on the "interface" part, but regarding the media-players, as I mentioned in the website, I also like A LOT the idea of modular players, possibly based on individual components and pipelines.
There are some similar projects that I mentioned like gstreamer (http://www.gstreamer.net) or NMM (http://www.networkmultimedia.org). Here is also a very interesting read (http://www.networkmultimedia.org/Status/MMBox/practicals/mmbox2002.pdf).
I think that it is quite important to separate at least the final A/V processing (volume/audio equalisation/time delay/video scaling/overlay...) as described in the Aduio/Video Sink nodes in the architecture section (http://avux.sourceforge.net/architecture.html). This is because ALL applications will use this type of processing and ensuring some sort of uniformity is quite important. I do not want the media-player to handle equalisation or volume control and then having duplicate controls at the end of the chain.
At the end of a day a media-player is only a demultiplex+codec, but it might include additional bits and pieces. In particular tehre are some A/V filters are used by some apps but not by others (like deinterlacing). In this case some players might incorporate those modules OR the player itself might be based on a modular structure. I think that it should be up to the end-user to decide what to use (even though I strongly favour the modular player idea).
I think that it would be nice to have an "interface" (in broad sense) to control ALL possible players (with suitable wrappers), modular or not. And to just take care of the a/v stream at the end of the chain (final processing + output to various devices).
Building a modular player is a difficult task and a major project in itself. On the other side the "interface" poses already LOTS of issues to be addressed. For instance what sort of table structure/fields are most suited to described ALL the multimedia files (including pictures, games, hyperlinks...)? What are the XML standards to be implemented for the communication between the ML layer and the GUI in order to "describe" the interface? What are the XML standards to describe available media-palyers/external apps, their services and their controls (think of a way to describe the functionality of a web browser and an mp3 player)? Shall we introduce "advanced features" like PiP, news tickers...? What type of event notification mechanism shall we use between "interface" components?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
At this stage I am still gathering ideas about the architecture and putting down some drafts, please feel free to make suggestions.
From an email I just received:
I am setting up a htpc, so this is something that I have put alot of thought into. Your idea is similar enough to mine that I have set your project page as my home page (as opposed to the blank page that I have had for the past year)
It seems to me that a successful implementation of such an environment may manifest itself in such a manner.
Your webpage says that you are not trying to re-invent the wheel, however, I think that the ability to
re-invent the wheel is a necessity, albeit, a
possibility that does not need to be exploited.
I imagine an environment which is made up of four categories of plugins. It should be possible to run any of these plugins simultaneously, and even run it's output into the input of another plugin in the same category.
Input - any variety of inputs... tv tuner, disk reader, internet transport, analog audio, keyboard, oystick, nintendo power glove... all it does is take the input, and pass it to the decode plugins, encapsulated in
a standardized transport for all of them.
Decode - takes input from the input plugins, and formats it so that it can be processed... a standard way of presenting video to the processors, a standard way of presenting audio to the processors,
a standard way of presenting user input...
Processing Plugins that can work independant of the data source, deinterlacers, dsp filters, databases, anything you can think of that can be done to data in realtime, should be able to be written as a processing plugin.
Output
The data gets passed to the output filters in the same format thay hit the processing filters. And is then formatted by these plugins for any imaginable output device. Direct sound, OSS, OpenGl, Directx, DirectFB, Writing directly to the video memory in 320x200 8 bit mode, blinking lights on your keyboard, and a disk writer, which might get it's data from another output codec, which would encode it into divx,mp3,wav,etc.
A sample of how I envision this may work in one situation would be..
Input from a tv tuner card.
The V4L Tv tuner input plugin grabs the data from V4L and wraps it up to send to the V4L Decoder. The V4L decoder takes the data and splits it into a seperate standard video and audio streams. The video stream gets run through a deinterlacer, a
noise filter, and a timeshifter. The audio stream gets sent to a equalizer, and 3d effect dsp.
Both the video stream is sent to the DirectFB output, the audio stream is sent to OSS. The video and audio streams are also sent to the Mpeg4 encoder plugin, which outputs to the disk writer plugin and the mpeg4 server plugin.
The user watches tv while it is recorded, and streamed out over the internet. Switching formats is easy, adding any kind of input you want is easy.
The interface would also be a processing plugin. Perhaps you could hit a button on a remote, which gets caught by a serial reciever, which gets sent to the serial input plugin, which goes to the IR decoder plugin, which goes to the Flashy interface processing plugin, which querys the database plugin (which gets the database from maybe the disk reader, or the http input plugin), and returns a file location which goes to the disk reader input plugin, which gets the next mp3, and sends it to the mp3 decoder plugin, which goes to the 3d effect dsp processing plugin, then to the OSS output plugin. The Flashy interface processing plugin gets sent to the Direct FB output plugin.
Overlaying video and audio streams could be configured in their respective output plugins.
Sorry, that was long.
Thanks a lot for your comments. very interesting.
In fact it is very, very similar to what I have in mind. You are going a bit deeper though that I first intended.
To begin with I would separate the "interface" (intended in a broad sense, not just the GUI, and including low level user interfaces, multimedia DB, file management utilities, scheduler, info grabbers, epg and other typical HTPC utilities), from the media players themself.
Both the "interface" and the players can probably be implemented in a plug-in system similar to the one you descibe but the separation is mostly justifed by the type of I/O stream between components. One thing is to send a synchronized A/V stream between two components and another is to send a simple xml string.
For a full blown HTPC the "interface" is quite complex as it includes several components itself (components that are sometimes "fixed" like the multimedia DB, and sometime "swappable" like the info grabbers or low level UI) and it should be as independent as possible from the players actually used to process the A/V stream.
I was focusing mostly on the "interface" part, but regarding the media-players, as I mentioned in the website, I also like A LOT the idea of modular players, possibly based on individual components and pipelines.
There are some similar projects that I mentioned like gstreamer (http://www.gstreamer.net) or NMM (http://www.networkmultimedia.org). Here is also a very interesting read (http://www.networkmultimedia.org/Status/MMBox/practicals/mmbox2002.pdf).
I think that it is quite important to separate at least the final A/V processing (volume/audio equalisation/time delay/video scaling/overlay...) as described in the Aduio/Video Sink nodes in the architecture section (http://avux.sourceforge.net/architecture.html). This is because ALL applications will use this type of processing and ensuring some sort of uniformity is quite important. I do not want the media-player to handle equalisation or volume control and then having duplicate controls at the end of the chain.
At the end of a day a media-player is only a demultiplex+codec, but it might include additional bits and pieces. In particular tehre are some A/V filters are used by some apps but not by others (like deinterlacing). In this case some players might incorporate those modules OR the player itself might be based on a modular structure. I think that it should be up to the end-user to decide what to use (even though I strongly favour the modular player idea).
I think that it would be nice to have an "interface" (in broad sense) to control ALL possible players (with suitable wrappers), modular or not. And to just take care of the a/v stream at the end of the chain (final processing + output to various devices).
Building a modular player is a difficult task and a major project in itself. On the other side the "interface" poses already LOTS of issues to be addressed. For instance what sort of table structure/fields are most suited to described ALL the multimedia files (including pictures, games, hyperlinks...)? What are the XML standards to be implemented for the communication between the ML layer and the GUI in order to "describe" the interface? What are the XML standards to describe available media-palyers/external apps, their services and their controls (think of a way to describe the functionality of a web browser and an mp3 player)? Shall we introduce "advanced features" like PiP, news tickers...? What type of event notification mechanism shall we use between "interface" components?