Contribution: CreateObject ImportResource etc

2006-02-01
2013-05-30
  • Nektarios K. Papadopoulos

    Hi Gena, Sergey and all,

    congratulations on the great project and thank you for the opportunity to work with it.
    On behalf of inAccessNetworks[1] and for the needs of the ePerSpace project [2], I've implemented some missing UPnP actions for MediaTomb , namely:

    CreateObject
    DestroyObject
    ImportResource
    GetTransferProgress
    StopTransferResource

    The code is complete but not extensively tested. However there are some issues:
    - It works with a patched version of libupnp, since I use the UpnpHttpXXXXX functions provided, but I had to add:
      + querying transfer progress
      + canceling a transfer
      + proxy support
    - I've only tested it with sqlite. Although, I didn't do anything sqlite specific.
    - I'm not sure if I broke anything. I mainly tested UPnP functionality. Neither WebUI nor javascript
    - I've changed the behavior of removeObject in ContentManager. Empty containers are not automatically removed.
    - Currently ImportResource stores the whole content in memory before writing to file system. This is what libupnp provides and can be a problem on large imports.
    - The meta-data of newly imported resources are based on the information provided in CreateObject (e.g. no info extraction after the file is downloaded). I guess this is not hard to fix.
    - I've made the directory to place uploaded content configurable, but I didn't update tomb-install script appropriately.

    Additionally, I'm sure that my code will need some refactoring and some other points we can discuss after I post the code.

    Please let me know if you are interested in these code and advise on the proper way to submit it.
    I know that the Patches area in the sourceforge project page is a good place, but I don't see any previous postings there.
    The modifications I made affect several files and I had to add a couple of new files. A single patch file might be hard to read. OTOH, it's not always obvious how to split it (per file ?, per functionality ?). Let me know what you prefer.
    Would a `diff -uirE` be OK ?

    Looking forward to work with you on merging my code in Mediatomb trunk.

    Best Regards,
    Nektarios K. Papadopoulos

    ---
    [1] http://www.inaccessnetworks.com/
    [2] http://www.ist-eperspace.org/

     
    • Jin

      Jin - 2006-02-01

      Hi!

      Well, first of all - cool! :) Those features indeed sound very interesting and provide useful functionality, so - thank you for the contribution! :)

      First question regarding libupnp: you are saying that it works with a patched version of the UPnP SDK, are those patches already available online or will you provide them separately? This of course adds some complexity - those who want to use the new features will have to patch their libupnp. We plan to integrate libupnp in our source tree in order to simplify all those things, let's see if we can already do this step for the next release.

      Regarding sqlite: I assume you extended functionality of the ContentManager object which uses Storage functions to talk to the database. Actually I am not sure if you had to do anything there at all - assuming that you first save the file on disk you can then simply add it to the database using the existing functions. If that is how you did it I see no problem regarding database interaction - should not matter if sqlite3 or mysql is being used, that's handled by the lower layers.

      Regarding removeObject - was there a specific reason why you adapted the behaviour, so empty containers are not removed automatically? I remember we added this, because when deleting content in PC-Directory the whole structure or virtual containers was left untouched, which of course was not very nice. There are probably other ways to solve this, I just would like to know the issue that you encountered so we can take it into account.

      Storing content in memory: this can indeed be a problem, MediaTomb was ported to the NSLU2 running openslug and I also have a version running on a mipsel platform; actually I was surprised that it worked because I think that we are quite memory consuming. So for embedded devices that would surely be an issue, however I think that we can improve on that in the future. Regarding your feature - once we integrate libupnp into the MediaTomb source tree, we can probably find a solution for that problem as well.

      Regarding metadata: yes, I think that would be easy to fix. Probably a good way would be to take the metadata that was provided, and in addition parse the file in order to fill in any possible gaps.

      Regarding tomb-install: we anyway plan to move away from that script, we decided that the server itself should take care of the installation. This will not only free us from the python dependency, but also make life easier for our users.
      A good thing would be to make the functionality that you implemented configurable via the config.xml file (to be able to enable or disable it - security reasons), but that should not be too hard.

      Now regarding the merge:
      quite a few changes and adaptions have been made, but I think that they mostly concern lower layers. Gena has improved data import speed and did some other adaptions regarding performance. Some changes were also made to the Strings module; what I am trying to say is: I am quite sure that it will not be possible to simply apply patches to the current tree. I think the best way would be, if you do a "make distclean" on your modified tree and send me the tarball/put it online; once our current code becomes release-ready we will integrate yours. We then will do a full test cycle, so in the end we should have a release which contains your contribution.

      Please tell us if this is ok for you, and also how you would like to appear in the contributor list (Name/E-Mail/Company name/whatever).

      Once again - thanks for helping us to improve MediaTomb! :)

      Kind regards,
      Jin

       
      • Nektarios K. Papadopoulos

        Hi,

        regardind libupnp: I'll provide the patches separately. I saw that you already distribute a patched version, so I prefered to patch libupnp than reimplement thigs.

        regarding sqlite: actually I did some changes at the sql_storage layer. Most changes are for the removeObject issue see below. Another minor change is to store the location for CdsContainer objects also. Now, a Contorl Point can set the title of the Object using CreateObject. In order to avoid handling problems with characters not supported by the filesystem, I ignore the title and use filenames based on objectID. E.g. CdsItem.33. So I need to know the location of CdsContainers explicitly.

        regarding removeObject: I understand why this was done in the first place. I set objects created through CreateObject/ImportResource to be restricted=false. And I actually remove the files/directories when DeleteObject is called. I felt that such permanent actions should be done only if explicitly requested.

        regarding content in memory: I know! my target device is an embedded one too. I just didn't have the time. It's at the top of my TODO list :-)

        regarding metadata: ok thanks

        regarding tombinstall: ok. Python was an overhead for my embedded device too.

        Regarding your comment about how I add files. Unfortunately, since in UPnP you must first do a CreateObject giving the meta-data and no content, I create the CdsObject (in the database) but with no real file in the filesystem. Then, afte ImportResource, I have the file on the file-system but the CdsObject is already created. I didn't find a way to effectively use the existing functions for this situation.

        Regarding the merge. I placed the tarball as requested here:

        http://skylab.inaccessnetworks.com/misc-downloads/mediatomb-0.8.1-ImportResource.tar.gz

        I can also add a patch against mediatomb 0.8.1 at sourceforge.

        Also you can find the tarball of the patched libupnp (based on CVS) here:

        http://skylab.inaccessnetworks.com/misc-downloads/libupnp-mediatomb-ImportResource.tar.gz

        Have you considered using the CVS services of sourceforge ? I could then help merging before your code is release-ready.

        Finally, regarding the contribution list, I'd like:
        inAccess Networks, Nektarios K. Papadopoulos <npapadop _at_ inaccessnetworks _dot_ com>

        I actually include the above as a Copyright notice in the 4 new source files if you don't mind.

        Let me know if you find anything wrong or confusing in my code.

        Thanks again,
        Nektarios

         
    • Jin

      Jin - 2006-02-01

      Hi,

      I got both tar files, thanks!
      Regarding character sets: well, we could use the same approach as on data import. We have an option that indicates what charset the filesystem is using, internally we work with UTF-8 (that's also specced for UPnP), so data from the controlpoint is coming in UTF-8 as well; when working with the filesystem we can then just simply convert back from UTF-8 to the filesystem charset which is specified in the config file. Allthough there is one catch to this: the titles in the database do not have to be unique, this can be a problem on the filesystem. Well, anyway, I first have to look at everything :) So will not say too much now without fully understanding the problems you encountered.

      Regarding adding files: I was not aware of that, so now I understand the problem. I think we could use the function updateObject, I'll have to look into it.

      Regarding the patch: you can also submit a patch on sourceforge if you wish, maybe someone is interested to check it out before the release.

      Regarding CVS... in the beginning of the project we gave this a thought, but we really prefer subversion. Actually I am not sure if subversion is already or maybe will become available on SF, I think it's not yet there, right?

      Anyway, I hope I will find some time soon to look at the new code, but I have to admit that the next release will still take a while. We started to rewrite the UI (which in my opinion was not the best decision at this time ;) but now too much has changed overall in the code, so there is no way back :)

      I'll keep you posted, but please understand that it can really take a while :)

      Thanks!
      Jin

       
      • Nektarios K. Papadopoulos

        Hi,

        Regarding character sets, filenames etc: I didn't encountered any problems. I proactively avoided them ;-)

        I'll submit the mediatomb patch tomorrow. Where do you think is most appropriate to submit the libupnp patch? Here or at libupnp ?

        I prefer subversion too :-). For sourceforge, it is in beta use now. It is scheduled for full deployment end of February.

        Looking forward for the release.

        cheers
        nek

         
        • Nektarios K. Papadopoulos

          Hi,
          sorry for the delay on submiting the patch. I just did [1]

          In the mean time, I did include a fix for the tomb-install script in this patch.

          I've also submited a minor bug-fix regarding the <udn/> configuration element [2]

          cheers,
          nek

          ---
          [1]https://sourceforge.net/tracker/index.php?func=detail&aid=1425429&group_id=129766&atid=715782
          [2]https://sourceforge.net/tracker/index.php?func=detail&aid=1425424&group_id=129766&atid=715782

           
          • Jin

            Jin - 2006-02-06

            Hi,
            thanks for the fixes!

            Now Gena and me have some work ahead of us to integrate everything and prepare the next release :)

            BTW, interesting device you got (I checked out the PDF) :)

            Regards,
            Jin

             
    • Jin

      Jin - 2006-02-03

      I think libupnp would be the more appropriate place, however you can submit it there and here as well - for now your mediatomb patch is the only software that uses those libupnp extensions :)

      just wondering, on what embedded device are you going to use the server?

      greetings,
      jin

       
      • Nektarios K. Papadopoulos

        For the moment, I just submitted a patch at libupnp [1].

        We're currently using it in inAccess MRG-110 [2]. An ARM based Multiservice Residential Gateway running GNU/Linux.

        ---
        [1]https://sourceforge.net/tracker/index.php?func=detail&aid=1425253&group_id=7189&atid=307189

        [2]http://www.inaccessnetworks.com/ian/products/reference_designs/MRG-110-PB

         

Log in to post a comment.