tgtl-devel Mailing List for The Game Texture Loader
Brought to you by:
bobvodka
You can subscribe to this list here.
2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
(2) |
Sep
(1) |
Oct
|
Nov
|
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2008 |
Jan
|
Feb
(2) |
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: GTL D. D. M. L. <tgt...@li...> - 2008-03-25 01:06:59
|
Hi there, while writing a CMakeLists.txt for GTL I tried a few things to create something that could automatically create a binary distribution package for all possible runtime libraries and such. While things look promising I discovered some problems I'd like to discuss: 1. PhysFs is part of GTL. This means when building GTL it has to link to physfs somehow. Unfortunately this makes things complicated because we have to choose a version of physfs (static, dynamic). This sometimes leads to conflicts with the physfs version the user application links in. In my opinion physfs should not be part of GTL. It would be better to use the user definable reader/seeker stuff to realize physfs support. This way all code knowing something about physfs can reside outside of GTL. The current API for physfs could be realized by inlined converter code in GameTextureLoader.hpp. 2. STL and boost types in GTL interface. Currently boost::function is used whenever the user should be able to supply code called by GTL. std::string is used everywhere too. There are two problems with that. The first is that user code has to use the exact same version of the stl and boost. This is not such a big problem alone. It just makes it harder for the user and a binary GTL package has to support more combinations of stl and runtime. What's even more problematic is the second thing. When user code uses special memory management stuff things can get really nasty. When an object allocated by the user code on its custom heap is deallocated by the dll using the runtime default heap it messes things up. This offen happens under the hood at places where you wouldn't expect it. The solution to this problem is not to use these objects in a dll interface. This sounds more drastic than it seems. Internal use of std::string and boost::function can stay as is. Only exported functions should fall back to plain old const char * for strings and an interface for a custom file like this: class IFileHandle { public: virtual long Read(char * destination, unsigned long byteCount)=0; }; Image* LoadTexture( IFileHandle & file, const char * name); Again backward compatibility could be kept by supplying inlined versions that call the new versions like this: inline Image* LoadTexture(const std::string & name) { return LoadTexture(name.c_str()); } The same applies to the boost::function stuff. class CDelegateFileHandle : public IFileHandle { public: virtual long Read(char * destination, unsigned long byteCount) { _loader(destination, byteCount); } LoaderFunc_t _loader; }; As the interface of GTL is actually quite simple these changes would not be that much work I think. 3. Error reporting by exception. This is something similar to point 2. Throwing exceptions across dll boundaries works only if everything (runtime version, compiler settings, stl) is the same in the dll and the user code. This is even easier to solve. As far as I can see there is currently just one exception thrown. I would catch this inside the dll and let the error cross the dll boundary in another way. Again backwards compatibility might be possible be supplying a inlined wrapper that detects the error and throws the exception outside the dll. 4. The destructor of Image is not protected. This will also wrack havoc when a user calls delete img and the memory heaps in GTL and user code differ. An elegant solution for this might be to override the class delete operator of Image and call FreeImage in there. Of course FreeImage itself has to call the global delete operator then. 5. There are declarations in GameTextureLoader.hpp that don't seem to belong there. One example is struct Imageinfo. It's not needed by the dll interface. The solution would be to move this stuff to a separate header file. ------------------ Hey. Seems like I wrote a lot of stuff. While all this seems like some work to do GTL would be much more robust and allow the user code to customize its memory management. Actually this is what a lot of games do. In theory with these changes at least a dll build of GTL might be used by several compilers on windows. What do you think? Markus |
From: GTL D. D. M. L. <tgt...@li...> - 2008-02-25 12:52:06
|
Hey, I noticed you making a few SVN changes last night, I was a tad suprised *chuckles* Sure, adding CMakeList.txt to the current GTL distro would be good, anything to improve the usability. As for the rest of the reply, woah.. blast from the past *chuckles* Infact, GTL3 has been pretty much 'done' for Win32, at least within my current "vision" of the code and exists in SVN @ http://phantomsoftware-svn.cvsdude.com/gametextureloader3 or https://phantomsoftware-browse.cvsdude.com/wsvn/wsvn/gametextureloader3/? for WebSVN. Right now, it's pretty much Win32 only for two reasons; 1) Win32 message queue to hand off work to the worker threads 2) Win32 file reading routines; these are needed for Async reading and I've also used them for non-async block loading. Infact, the Async routine has alot of Win32 dependancy stuff in it simply due to how it works, however this is all hidden away in a Win32.cpp file. I'm currently looking at threadpool to see if that can be used to replace the platform specific message queue stuff, from an inital look it does look promising. I'm not sure what, if anything, can be done with the platform specific asyic io. Right now a work unit is loading the whole file, either successfully or unsuccessfully, which is indeed handed off to a thread to process. The reason behind this is that the whole image is read in in one block and then processed (in a single pass over the data, which includes TGA RLE decompression and changing the origin). The host can query for state in two ways; 1) polling as before 2) 2 call back functions; one for 'success' and one for 'failure'. Right now, these are global for the whole lib and while it can be changed it will only affect latter load calls. The idea of notify as mipmaps are loaded is a nice idea, however it reverse loading of mipmaps with current formats is pretty much impossible; the only format with mipmaps is DDS files and those are stored largest=>smallest, so to load the smallest mipmap you need to read passed all the rest anyway. - Rob On 25/02/2008, GTL Development Discussion Mailing List < tgt...@li...> wrote: > > Hi there, > > for my own needs I wrote a CMakeLists.txt file to build GTL with cmake. > If you are interested I can contribute it. It could even be used to > generate MSVC projects for every insane combination of static/dynamic > build and runtime. I really love using cmake for my cross platform > projects. PhysFS seems to have to be using it too. > > > So now to your thought about GTL 3.0. Somehow I must have overlooked > your email. So this reply is a bit late :-) > > Rob Jones wrote: > > > > > > So, I'm not dead, which is a good thing J > > > > > > > > I've put some time today into considering how the Async IO portion of > > GTL3.0 can be done. Right now nothing is set in stone as these are kind > > of first ideas, but I thought I'd throw them out there. > > > > > > > > As the idea is to hand off data for loading and processing and return as > > soon as possible the system is going to have to use thread; while I did > > toy with using Boost::thread to get around platform specific code the > > concerns about using extra compiled libs like boost have been noted. > > Also, for the win32 solution we need the thread id so that isn't really > > workable. > > > > > > > > The basic idea as it stands is to create a pool of worker threads > > (either compile time or run time specified, if the latter then a > > GTL::Init() function is going to have to exist, I'm thinking this is a > > better solution as it also gets rid of some of the static hackery which > > is in GTL2.0 with regards to the filter registering system); each thread > > then blocks and waits on a message which contains the data package to > > process. When they have data to process they process it, setting flags > > as they go, and then when done check for more work, if no more work > > exists they go back into a blocking state and wait for more data. This > > stops the threads from chewing up resources by constantly polling for > > data. When a thread is allocated data its idea is shuffled to the end of > > a queue so data is assigned to free threads as often as possible. > > > This sounds like the thread pool pattern. An interesting point would be > what a work unit is. For example: Will a thread be given the task to > read the header and is returned to the pool so another thread is given > the task to read the rest? Or is one thread dedicated to one image until > everything is loaded successfully? > > > > > > On Win32 this would work via the built in messaging system via > > GetMessage(). The reason we need the thread id is so data can be posted > > to the threads message queue. On Linux and OS X it seems that the > > message queue (mq_open et al) system can be bent to a similar task, > > however in this case we won't need to cycle threads as I think the first > > free thread will pick data off the queue. > > I don't think it is a good idea to become that platform specific. > Boost::thread has condition variables. A thread can wait for a condition > to become true and will not consume any CPU cycles. May be this could be > useful too: > http://threadpool.sourceforge.net/ > > > > > I'd also like to allow for pulling data from remote sites via http, for > > this libcurl looks to be the best answer; as the download functions are > > blocking (iirc, it's been a while since I used it) this fits in with the > > above model perfectly. > > > > > > > > The idea of 'flags' was mentioned above; depending on the final > > implementation I can see the following flags being needed; > > > > - Queued – in the processing queue > > > > - Processing – data is being processed but nothing is available > > yet > > > > - Header – header data is done, so things like size, format etc > > have been decoded > > > > - Completed – all data is downloaded and ready to use > > > > - Error flags – various error flags depending on failure states > > > > > > I think a very important point is to support loading of mipmap levels in > reverse order so new textures can be displayed in lowres while still > loading. Maybe there should be some flag when a miplevel is done. > > > > The front API I'd like to keep pretty much the same as before where > > possible, basically added a flag for Async requests to the functions > > which will allow it (which would exclude PhysFS based one as afaik you > > can't do Async IO with it). > Good idea. But how do you want to report loading progress to user code? > Should TGTL be polled until something happens? Should there be some kind > of Run method that calls registered handlers? > > > > > Anyways, that's where I am with things in my head right now. > > > > One thing I want to do is keep the backend for this as data/processing > > agnostic as possible as I plan to write a library for loading sound data > > as well and I figure reusing the backend would be a good idea ;) > > > > > > > > Rob > > Markus > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > tgtl-devel mailing list > tgt...@li... > https://lists.sourceforge.net/lists/listinfo/tgtl-devel > |
From: GTL D. D. M. L. <tgt...@li...> - 2008-02-25 03:13:16
|
Hi there, for my own needs I wrote a CMakeLists.txt file to build GTL with cmake. If you are interested I can contribute it. It could even be used to generate MSVC projects for every insane combination of static/dynamic build and runtime. I really love using cmake for my cross platform projects. PhysFS seems to have to be using it too. So now to your thought about GTL 3.0. Somehow I must have overlooked your email. So this reply is a bit late :-) Rob Jones wrote: > > > So, I’m not dead, which is a good thing J > > > > I’ve put some time today into considering how the Async IO portion of > GTL3.0 can be done. Right now nothing is set in stone as these are kind > of first ideas, but I thought I’d throw them out there. > > > > As the idea is to hand off data for loading and processing and return as > soon as possible the system is going to have to use thread; while I did > toy with using Boost::thread to get around platform specific code the > concerns about using extra compiled libs like boost have been noted. > Also, for the win32 solution we need the thread id so that isn’t really > workable. > > > > The basic idea as it stands is to create a pool of worker threads > (either compile time or run time specified, if the latter then a > GTL::Init() function is going to have to exist, I’m thinking this is a > better solution as it also gets rid of some of the static hackery which > is in GTL2.0 with regards to the filter registering system); each thread > then blocks and waits on a message which contains the data package to > process. When they have data to process they process it, setting flags > as they go, and then when done check for more work, if no more work > exists they go back into a blocking state and wait for more data. This > stops the threads from chewing up resources by constantly polling for > data. When a thread is allocated data its idea is shuffled to the end of > a queue so data is assigned to free threads as often as possible. > This sounds like the thread pool pattern. An interesting point would be what a work unit is. For example: Will a thread be given the task to read the header and is returned to the pool so another thread is given the task to read the rest? Or is one thread dedicated to one image until everything is loaded successfully? > > On Win32 this would work via the built in messaging system via > GetMessage(). The reason we need the thread id is so data can be posted > to the threads message queue. On Linux and OS X it seems that the > message queue (mq_open et al) system can be bent to a similar task, > however in this case we won’t need to cycle threads as I think the first > free thread will pick data off the queue. I don't think it is a good idea to become that platform specific. Boost::thread has condition variables. A thread can wait for a condition to become true and will not consume any CPU cycles. May be this could be useful too: http://threadpool.sourceforge.net/ > > I’d also like to allow for pulling data from remote sites via http, for > this libcurl looks to be the best answer; as the download functions are > blocking (iirc, it’s been a while since I used it) this fits in with the > above model perfectly. > > > > The idea of ‘flags’ was mentioned above; depending on the final > implementation I can see the following flags being needed; > > - Queued – in the processing queue > > - Processing – data is being processed but nothing is available > yet > > - Header – header data is done, so things like size, format etc > have been decoded > > - Completed – all data is downloaded and ready to use > > - Error flags – various error flags depending on failure states > > I think a very important point is to support loading of mipmap levels in reverse order so new textures can be displayed in lowres while still loading. Maybe there should be some flag when a miplevel is done. > The front API I’d like to keep pretty much the same as before where > possible, basically added a flag for Async requests to the functions > which will allow it (which would exclude PhysFS based one as afaik you > can’t do Async IO with it). Good idea. But how do you want to report loading progress to user code? Should TGTL be polled until something happens? Should there be some kind of Run method that calls registered handlers? > > Anyways, that’s where I am with things in my head right now. > > One thing I want to do is keep the backend for this as data/processing > agnostic as possible as I plan to write a library for loading sound data > as well and I figure reusing the backend would be a good idea ;) > > > > Rob Markus |
From: Rob J. <rob...@gm...> - 2007-09-11 16:11:55
|
So, I'm not dead, which is a good thing J I've put some time today into considering how the Async IO portion of GTL3.0 can be done. Right now nothing is set in stone as these are kind of first ideas, but I thought I'd throw them out there. As the idea is to hand off data for loading and processing and return as soon as possible the system is going to have to use thread; while I did toy with using Boost::thread to get around platform specific code the concerns about using extra compiled libs like boost have been noted. Also, for the win32 solution we need the thread id so that isn't really workable. The basic idea as it stands is to create a pool of worker threads (either compile time or run time specified, if the latter then a GTL::Init() function is going to have to exist, I'm thinking this is a better solution as it also gets rid of some of the static hackery which is in GTL2.0 with regards to the filter registering system); each thread then blocks and waits on a message which contains the data package to process. When they have data to process they process it, setting flags as they go, and then when done check for more work, if no more work exists they go back into a blocking state and wait for more data. This stops the threads from chewing up resources by constantly polling for data. When a thread is allocated data its idea is shuffled to the end of a queue so data is assigned to free threads as often as possible. On Win32 this would work via the built in messaging system via GetMessage(). The reason we need the thread id is so data can be posted to the threads message queue. On Linux and OS X it seems that the message queue (mq_open et al) system can be bent to a similar task, however in this case we won't need to cycle threads as I think the first free thread will pick data off the queue. I'd also like to allow for pulling data from remote sites via http, for this libcurl looks to be the best answer; as the download functions are blocking (iirc, it's been a while since I used it) this fits in with the above model perfectly. The idea of 'flags' was mentioned above; depending on the final implementation I can see the following flags being needed; - Queued - in the processing queue - Processing - data is being processed but nothing is available yet - Header - header data is done, so things like size, format etc have been decoded - Completed - all data is downloaded and ready to use - Error flags - various error flags depending on failure states The front API I'd like to keep pretty much the same as before where possible, basically added a flag for Async requests to the functions which will allow it (which would exclude PhysFS based one as afaik you can't do Async IO with it). Anyways, that's where I am with things in my head right now. One thing I want to do is keep the backend for this as data/processing agnostic as possible as I plan to write a library for loading sound data as well and I figure reusing the backend would be a good idea ;) Rob |
From: Markus H. <mar...@gm...> - 2007-08-20 20:36:31
|
Robert Jones schrieb: > Hi, > > Firstly, sorry about the huge delay in replying to this; between my normal > ability to be distracted by 'good ideas', finding and starting work and my > email server falling off the internet I've not had a chance to reply. > > Anyways; thanks for all the fixes and work you did. I've now merged them > back into the main branch and, along with a rewrite of the PNG setup code > (your reading loop is still the same, I just made some changes so it could > handle paletted, grey and grey + alpha images), I've done a 2.2 release. > > Now, the bugs; > > I can't see any problem with your plan to fix problem which files which > can't be opened for writing. Indeed if I had remembered about this bug I'd > have fixed it myself before the 2.2 release (2.2.1 at some point soon I > guess then). > > I did however remember about what you wrote with regards to the iostreams > and boost usage; > > The iostreams got used for two reasons; one being my attraction to new > things and the other being I thought they could solve various design issues. > They did in a way but reintroduced a number along the road, so yes for the > next major version I agree that they should be removed. In fact, I don't > know how compatible they would be with the concept of adding async file > loading in the next revision (optional; so you can still load as now but > also an interface will exist to allow async loading.. I'm still working on > the details). > > As for kicking boost out completely, I'm not so convinced; while iostreams > is going to go until C++ gets fixed data sizes I'd much rather rely on > someone else doing the work for other platforms rather than myself (or > anyone else). Also, boost::shared_ptr<> is in the public interface for the > safe loading routines and while writing a smart pointer could be considered > trivial I'm not keen on reinventing that wheel; plus it just becomes another > interop problem as now the user needs to support whatever smart pointer we > come up with and potentially another, I'd much rather go with something more > standard (if TR1 was widely in use I'd consider dropping boost::shared_ptr > for the TR1 revision, at which point the question about the usefulness of > the fixed data sizes from boost alone could be called into question... in > fact, there is probably some logic to switching to TR1 for the next version > as boost supplies a lot of it, I'll have to try to remember to look into > that). > > My final reasoning is that I would hope that GTL would be used in a large > project and, imo, any C++ programmer who is even vaguely serious about using > the language should have boost available, in header form if not compiled. > > So, in summary, while I'm behind shunting out iostreams boost itself stays > for now, unless you have some other convincing reasons you can throw at me? My main problem with boost was that it is such a huge thing. For small projects that want to use GTL just to get some textures loaded this is quite a lot of stuff to download. Furthermore a lot of game developers are quite conservative when using external libraries. When I introduced boost to my co-workers they thought I was joking to use a library with 20MB headers and about 1Gig of libs. On the other hand these people are probably not the ones using GTL as they will stick with the standard windows+directx setup anyway. Additionally GTL doesn't need to link to any boost lib so the headers are enough when iostreams are removed. So I think I can easily live with your opinion. > With regards to the next major version I'll try to get some 'grand ideas' > down on paper soon and post them to the list. Sounds interesting. One thing I could really need is image resizing and format conversion. Did you plan support for this stuff? Collada FX does support specifying texture size and format that differ from the image file used. It's quite a mess to do that using gluScaleImage as even version GLU 1.3 doesn't support all texture formats of opengl 2.0. But I'm not sure if it's not better to just skip this feature and leave image conversion to external applications as GTL was written to provide features actually needed by game engines and not stuff that hardly anyone might need. Regards, Markus |
From: Robert J. <rob...@gm...> - 2007-08-20 05:22:13
|
Hi, Firstly, sorry about the huge delay in replying to this; between my normal ability to be distracted by 'good ideas', finding and starting work and my email server falling off the internet I've not had a chance to reply. Anyways; thanks for all the fixes and work you did. I've now merged them back into the main branch and, along with a rewrite of the PNG setup code (your reading loop is still the same, I just made some changes so it could handle paletted, grey and grey + alpha images), I've done a 2.2 release. Now, the bugs; I can't see any problem with your plan to fix problem which files which can't be opened for writing. Indeed if I had remembered about this bug I'd have fixed it myself before the 2.2 release (2.2.1 at some point soon I guess then). I did however remember about what you wrote with regards to the iostreams and boost usage; The iostreams got used for two reasons; one being my attraction to new things and the other being I thought they could solve various design issues. They did in a way but reintroduced a number along the road, so yes for the next major version I agree that they should be removed. In fact, I don't know how compatible they would be with the concept of adding async file loading in the next revision (optional; so you can still load as now but also an interface will exist to allow async loading.. I'm still working on the details). As for kicking boost out completely, I'm not so convinced; while iostreams is going to go until C++ gets fixed data sizes I'd much rather rely on someone else doing the work for other platforms rather than myself (or anyone else). Also, boost::shared_ptr<> is in the public interface for the safe loading routines and while writing a smart pointer could be considered trivial I'm not keen on reinventing that wheel; plus it just becomes another interop problem as now the user needs to support whatever smart pointer we come up with and potentially another, I'd much rather go with something more standard (if TR1 was widely in use I'd consider dropping boost::shared_ptr for the TR1 revision, at which point the question about the usefulness of the fixed data sizes from boost alone could be called into question... in fact, there is probably some logic to switching to TR1 for the next version as boost supplies a lot of it, I'll have to try to remember to look into that). My final reasoning is that I would hope that GTL would be used in a large project and, imo, any C++ programmer who is even vaguely serious about using the language should have boost available, in header form if not compiled. So, in summary, while I'm behind shunting out iostreams boost itself stays for now, unless you have some other convincing reasons you can throw at me? With regards to the next major version I'll try to get some 'grand ideas' down on paper soon and post them to the list. -----Original Message----- From: tgt...@li... [mailto:tgt...@li...] On Behalf Of Markus Henschel Sent: 27 July 2007 21:24 To: tgt...@li... Subject: [tgtl-devel] Hello and more Hi there, as you have noticed I'm through with my patches in the bug fixing branch. There is one "bug" left I would like to discuss first: Bug: Loading a texture from a file fails when the file cannot be opened for writing (read only file system!). While it would be easy to just change the io mode when opening the file my solution would be to just remove the whole writing and seeking stuff and just work with boost source like devices. I didn't see writing being used anywhere and I think I removed seeking from all the filters. Do you see any problems with that? This leads me to another point. The whole device concept with boost::iostreams seems a little bit over engineered to me. While I think boost is a great thing and I advice it's usage in all bigger C++ projects I think for GTL the cost is bigger than it's value. I think it would be much easier for users of GTL not being forced to install additional dependencies to use it. The device interface could be implemented just as easy without boost::iostreams. The fixed size data types could be easyly moved to custom typedefs too. So for the next major release I think it would be a good idea to kick out boost and simplify the device concept. Do you agree? Have a nice weekend! Markus ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ tgtl-devel mailing list tgt...@li... https://lists.sourceforge.net/lists/listinfo/tgtl-devel |
From: Markus H. <mar...@gm...> - 2007-07-27 20:24:21
|
Hi there, as you have noticed I'm through with my patches in the bug fixing branch. There is one "bug" left I would like to discuss first: Bug: Loading a texture from a file fails when the file cannot be opened for writing (read only file system!). While it would be easy to just change the io mode when opening the file my solution would be to just remove the whole writing and seeking stuff and just work with boost source like devices. I didn't see writing being used anywhere and I think I removed seeking from all the filters. Do you see any problems with that? This leads me to another point. The whole device concept with boost::iostreams seems a little bit over engineered to me. While I think boost is a great thing and I advice it's usage in all bigger C++ projects I think for GTL the cost is bigger than it's value. I think it would be much easier for users of GTL not being forced to install additional dependencies to use it. The device interface could be implemented just as easy without boost::iostreams. The fixed size data types could be easyly moved to custom typedefs too. So for the next major release I think it would be a good idea to kick out boost and simplify the device concept. Do you agree? Have a nice weekend! Markus |