Thread: [GD-Windows] Background loading thread
Brought to you by:
vexxed72
Menu
▾
▴
gamedevlists-windows
|
From: Ivan-Assen I. <iva...@gm...> - 2006-09-08 20:41:47
|
Hello, I'm in the process of reengineering our data loading process with the goal of moving as much as possible of the loading of art assets into a separate thread. Now, if all the ReadFile calls happen from a separate thread, devoted 100% to them, is there a good reason to inflict on myself considerable pain and suffering by using the overlapped I/O API? We have a perfectly good packfile/filesystem support which should be thread-safe for the most part (famous last words material, eh?) except for a few isolated places which can be mutexed out, and which will have to be thrown away to use overlapped API. I hope Windows has the common sense to switch away from a thread which is performing a lengthy synchronuous IO operation? What about a hypothetical other platform, that would happen be produced by the same OS vendor around the same OS kernel, would be in even greater need of offloading loading to another core, but wouldn't happen to be covered by NDAs? Does anyone have any related war stories to tell? Best regards, Assen |
|
From: Jan W. <ur...@st...> - 2006-09-08 20:51:30
|
Hello, > Now, if all the ReadFile calls happen from a separate thread, devoted > 100% to them, is there a good reason to inflict on myself considerable > pain and suffering by using the overlapped I/O API? > Does anyone have any related war stories to tell? I have done some looking into this topic; you may be interested in the resulting paper: http://www.stud.uni-karlsruhe.de/~urkt/articles/study_thesis.pdf . There is brief discussion of sync vs. async IO in Section 3.1. Questions, comments and discussion welcome! Best Regards Jan Wassenberg |
|
From: Mat N. (BUNGIE) <Mat...@mi...> - 2006-09-08 21:35:04
|
1. Why burn a thread to do synchronous I/O? You can do sync I/O on the same thread and have more opportunity to interleave reads and other similar logic. 2. True overlapped I/O (NO_BUFFERING, ReadFileScatter, and IOCompletionPorts to avoid the event handle) is about as optimal as you can get. And it's easy to interrupt and all that other fun stuff that you'll want eventually. 3. If you are going to mix huge sequential reads and streaming, you can use asynchronous I/O or eat the logic cost of reading smaller buffers manually. MSN -----Original Message----- From: gam...@li... [mailto:gam...@li...] On Behalf Of Ivan-Assen Ivanov Sent: Friday, September 08, 2006 1:42 PM To: gam...@li... Subject: [GD-Windows] Background loading thread Hello, I'm in the process of reengineering our data loading process with the goal of moving as much as possible of the loading of art assets into a separate thread. Now, if all the ReadFile calls happen from a separate thread, devoted 100% to them, is there a good reason to inflict on myself considerable pain and suffering by using the overlapped I/O API? We have a perfectly good packfile/filesystem support which should be thread-safe for the most part (famous last words material, eh?) except for a few isolated places which can be mutexed out, and which will have to be thrown away to use overlapped API. I hope Windows has the common sense to switch away from a thread which is performing a lengthy synchronuous IO operation? What about a hypothetical other platform, that would happen be produced by the same OS vendor around the same OS kernel, would be in even greater need of offloading loading to another core, but wouldn't happen to be covered by NDAs? Does anyone have any related war stories to tell? Best regards, Assen ------------------------------------------------------------------------ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D= 121642 _______________________________________________ Gamedevlists-windows mailing list Gam...@li... https://lists.sourceforge.net/lists/listinfo/gamedevlists-windows Archives: http://sourceforge.net/mailarchive/forum.php?forum_id=3D555 |
|
From: Mat N. (BUNGIE) <Mat...@mi...> - 2006-09-08 21:37:19
|
#1 should read as: If you are going to burn a thread on I/O, can use that thread more effectively with overlapped I/O. MSN -----Original Message----- From: gam...@li... [mailto:gam...@li...] On Behalf Of Mat Noguchi (BUNGIE) Sent: Friday, September 08, 2006 2:34 PM To: Game Development for MS Windows Subject: Re: [GD-Windows] Background loading thread 1. Why burn a thread to do synchronous I/O? You can do sync I/O on the same thread and have more opportunity to interleave reads and other similar logic. 2. True overlapped I/O (NO_BUFFERING, ReadFileScatter, and IOCompletionPorts to avoid the event handle) is about as optimal as you can get. And it's easy to interrupt and all that other fun stuff that you'll want eventually. 3. If you are going to mix huge sequential reads and streaming, you can use asynchronous I/O or eat the logic cost of reading smaller buffers manually. MSN -----Original Message----- From: gam...@li... [mailto:gam...@li...] On Behalf Of Ivan-Assen Ivanov Sent: Friday, September 08, 2006 1:42 PM To: gam...@li... Subject: [GD-Windows] Background loading thread Hello, I'm in the process of reengineering our data loading process with the goal of moving as much as possible of the loading of art assets into a separate thread. Now, if all the ReadFile calls happen from a separate thread, devoted 100% to them, is there a good reason to inflict on myself considerable pain and suffering by using the overlapped I/O API? We have a perfectly good packfile/filesystem support which should be thread-safe for the most part (famous last words material, eh?) except for a few isolated places which can be mutexed out, and which will have to be thrown away to use overlapped API. I hope Windows has the common sense to switch away from a thread which is performing a lengthy synchronuous IO operation? What about a hypothetical other platform, that would happen be produced by the same OS vendor around the same OS kernel, would be in even greater need of offloading loading to another core, but wouldn't happen to be covered by NDAs? Does anyone have any related war stories to tell? Best regards, Assen ------------------------------------------------------------------------ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D= 121642 _______________________________________________ Gamedevlists-windows mailing list Gam...@li... https://lists.sourceforge.net/lists/listinfo/gamedevlists-windows Archives: http://sourceforge.net/mailarchive/forum.php?forum_id=3D555 ------------------------------------------------------------------------ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D= 121642 _______________________________________________ Gamedevlists-windows mailing list Gam...@li... https://lists.sourceforge.net/lists/listinfo/gamedevlists-windows Archives: http://sourceforge.net/mailarchive/forum.php?forum_id=3D555 |
|
From: Jon W. <hp...@mi...> - 2006-09-08 23:16:43
|
Ivan-Assen Ivanov wrote:
> I hope Windows has the common sense to switch away from a thread which
> is performing a lengthy synchronuous IO operation?
Yes, it does.
The main gain in overlapped I/O is that the amount of device
parallelism, and the degree of completion parallelism, can be separated.
With ReadFile() in threads, you need to create as many threads as you
want to have simultaneous disk loads outstanding; with IOCP, you only
need to create as many threads as you want simultaneous completions. The
typical number for a hard disk is 4 or 5 for the former, but only one
for the latter (unless you have a massively parallel I/O subsystem,
which means you're not doing a typical game asset load thing anymore).
If you can pay the cost of the thread safety, and the stacks of the
threads, then you can go with some number of threads for your I/O needs.
Just don't call CreateThread() for each I/O operation...
Cheers,
/ h+
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X