Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Multi-Threading and DevIL

2007-06-15
2013-05-02
  • Hi,
    I have just integrated DevIL into my project and I think it is a great project. Thanks for your effort.

    My first impression - looking at the interface in <il/il.h> - I presume that DevIL is not multi-thread safe due to the need to bind images before calling a function. Is this correct?

    This seems to be unneccessary, since you also have all the functions where you provide a pointer to the ILimage. By adding the image handle to the interface functions the library could be made multi-thread safe without much effort.

    Regards,
    Jan

     
    • Dario Meloni
      Dario Meloni
      2007-06-15

      Most of the data passing between internal functions has been done with globals (such as the current image) and most of the loaders store even more info in global variables

       
    • Nicolas Weber
      Nicolas Weber
      2007-06-15

      As Dario pointed out, DevIL uses lots of global internally. You have to make sure that you put all your DevIL calls into a mutex guarded section, then you're fine -- this means, of course, that you can't load two images simultaneously.

       
    • Thanks for your replies.

       
    • Denton Woods
      Denton Woods
      2009-01-08

      I have seriously thought recently about making DevIL thread-safe like Jan mentioned.

       
    • nitro
      nitro
      2009-01-31

      What's the use case for a multi-threaded devil

      For things like mass-conversion of images and such you could just create a process pool and each process runs its own version of devil. Then you assign your tasks to processes which are currently free.
      Threading makes a lot of things a lot more difficult and it looks like making DevIL threadsafe is a really hard task which probably has just marginal benefits (unless there's some compelling usecase I cannot think of now).

       
    • ApellA
      ApellA
      2009-02-26

      Actually, it would be handy, if you had to load a large amount of images.
      Instead of loading them one by one, it would be nice to use the processing power of multiple processors, if present, to load, e.g. 4 images at the same time.
      The biggest bottleneck would still be the hard drive, tough, and I do not know if that is already the case now.
      If so, then multithreading wouldn't be interesting indeed.

       
    • Ben Campbell
      Ben Campbell
      2009-02-26

      I'm inclined to agree that the I/O speed (ie the harddrive) will far outweigh any processing time required to decode images in pretty much any image format you can name.
      It's also going to depend on how thread-safe the various dependent libraries are (libpng, libjpeg, whatever)...

      My gut feeling is that it's a lot of work for very little gain, with a lot of potential for really nasty little corner-case bugs and added complexity.
      Unless you've got a _really_ good reason to do it, I'd probably avoid it.

       
      • Denton Woods
        Denton Woods
        2009-02-26

        Some new formats, such as HD Photo, require quite a bit of processing time to load.  Also, some libraries read in chunks of image data and then decode, instead of reading in value by value, so they do not have to wait on the harddrive all of the time.  I have not done any profiling, but I believe that with multiple cores, loading a lot of images could be slightly faster.  Also, DevIL frequently has to do conversions after loading an image, since most people want their data in a certain format (such as RGBA).

        The other case that DevIL deals with is when you are loading images from a memory buffer, so there is no harddrive access.  You could definitely run lots of these parallel.  I know that there are multiple people using this feature.

        You may be right that there will be little gain with the possibility of introducing new bugs, and I will definitely think about it some more.

         
    • Ben Campbell
      Ben Campbell
      2009-02-26

      (just a little extra note, from a games development perspective)
      Multithreading is the big thing in games these days, as consoles these days (PS3, 360, Wii) generally have multicore processors and need to be given multiple tasks to keep them running at full capacity.
      I suspect this background is where a lot of the desire to make DevIL multithreaded comes from.

      On the consoles, because they are likely to be loading from slow slow slow optical media, there's actually less reason to be running multiple image decompressions in parallel. The bottleneck is always going to be getting the files off the disk rather than converting them into whatever internal texture format the hardware needs. Even this conversion is likely to be done offline anyway - the files on disk are likely to be a native hardware format, probably with compression to speed up the loading. And while stuff can be cached/installed to a hard drive on consoles that support this, my guess (and it is just a guess) is that they are _not_ superfast drives. They'll be the cheapest bulk-buy drives sony or microsoft can lay their hands on ;-)
      And on top of all this, I'm pretty sure all the console publishers balk and choke with horror at the merest whiff of open-source code ;-)
      (when I did console development, I remember people being paranoid about even using open source _tools_!)
      So I would drop any thought of consoles from any thought processes involved in the multi-threaded decision.

      Whether or not it's a win on PC is the question.

       
  • emymrin
    emymrin
    2009-10-15

    FYI, I added support for multithreading to DevIL library by going through all sources and prefixing all static variables' declarations with __declspec(thread). This allowed multiple threads to simultaneously load images.
    Of course this is Windows-only solution.