Menu
▾
▴
mingw-users
|
From: sasuke <uzu...@gm...> - 2012-08-20 08:26:32
|
Hi folks,
I have been trying to "build" a shared-dynamic library (DLL) on Windows for
a largish code base (around 800 CPP files which result in 800 .o files)
using the MingW toolset and seem to have hit upon a snag. The problem occurs
during the linking phase when a single large command is created which has
*all* the .o files on it. Something like:
g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll
I have tried out pretty much every free IDE out there on Windows and the
result is the same. The object files are created all right but the linking
step fails every time with some weird error message which ultimately boils
down to "too long argument list". For e.g. on Netbeans, the error I get is
"make[2]: execvp: g++: Bad file number" which as per my searching again
comes back to the long list of object files. Strangely enough, Visual studio
seems to work fine with the given project and successfully creates the DLL
even with the long command line.
I have tried pretty much every trick I knew of. Using the @file flag for g++
etc. but it still doesn't work. Is there *anyone* out there who has
successfully managed to create a DLL for a largish project using the MingW
toolset. Please let me know if there are any queries or additional
information needed. Any assistance/tips would be greatly appreciated!
TIA,
sasuke
--
View this message in context: http://old.nabble.com/Dealing-with-extremely-large-command-line-when-linking-files-tp34323030p34323030.html
Sent from the MinGW - User mailing list archive at Nabble.com.
|
|
From: LRN <lr...@gm...> - 2012-08-20 08:31:02
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20.08.2012 12:26, sasuke wrote: > > Hi folks, > > I have been trying to "build" a shared-dynamic library (DLL) on > Windows for a largish code base (around 800 CPP files which result > in 800 .o files) using the MingW toolset and seem to have hit upon > a snag. The problem occurs during the linking phase when a single > large command is created which has *all* the .o files on it. > Something like: > > g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll > Well, for C i would have used an intermediate library trick: compile some .o files into static .a libraries, then link these libraries together. Not sure if C++ supports such things. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJQMfW8AAoJEOs4Jb6SI2Cw6D8IALMeZX5p+MHSaXRdEgPkbkxt HYASIDXa0EjXq2ZHX6w6bsXIdFNMYfDzyPbdDCwB66sSbl+uUiepOZM1GinuA4Fo NktRuXVAeZe96l3yqVkf+A3sL1IboWzxIKomkc4oYNg0hSS8+Iq+QhHtUAQl47J0 8Y/3k+HQy42RwrrYtCSUtbWYrXXMvcKaqoNP/xHr8V1CBBIi2oo7l5cqvDN/IPYj MyQcBr2+vwQ38XdBsI+zVNzzQ2YOxqfUnpiOxl6nP6RRP+g3OYpeZ2fIBsp2s531 pvMF2SNgNhmWgZxAu3igQYwAXYcWr/QOPPlAOP8b3NY4vLoMRFdjKlxPrJwmEbU= =ue5s -----END PGP SIGNATURE----- |
|
From: sasuke <uzu...@gm...> - 2012-08-21 06:07:02
|
Thanks a lot for your replies folks! I'll try to address each of them below. L.R.N wrote: > > On 20.08.2012 12:26, sasuke wrote: >> >> <snip explanation> >> > Well, for C i would have used an intermediate library trick: compile > some .o files into static .a libraries, then link these libraries > together. Not sure if C++ supports such things. > This is what I initially had in mind i.e. merging the .o files created in batches of 100 and then firing the final command with the newly created 8 .o files (assuming there are 800 .o files originally). But then I realized that this method is way too involved and painful to automate. Plus I was really looking for the reason as to why is this giving me problems. Joe Burmeister wrote: > > Hi Sasuke, > > This is similar to what I hit a while ago, patch attached. > > Now in my case I couldn't switch to using the @ option file because the > make files where auto-generated and that would have been a lot more work > to switch to. > Instead, I increased the active command line limit, where I was hitting > it, from 8k to 64k. > Unless invoke from CreateProcess, the Windows command line only has a 8k > limit, only this isn't enforced everywhere, causing, strangeness..... > If the command line is called directly (i.e. not via cmd.exe) from > CreateProcess, you get the higher 64k limit. > > For you this may or may not help as your line may still be too long. It > would be easy to argue this is Windows fault not MinGW, but that won't > help you. > > > The way I see it, you can: > > * Use the @ option file, > http://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html > * Group stuff together into libs and link those (using a def file to > insure you still get the right DLL interface). > * Do incremental linking. > * Use MinGW on Linux to cross build your project. ;-) > > Hope this helps. :-) > Thanks Joe, I finally see the underlying issue. I really wasn't aware that there was another way of doing this and was banging my head thinking how was Visual Studio able to cope up with the large argument size. It's kind of weird that pretty much *every* free IDE on Windows out there is broken in this regard. Eclipse CDT/ Netbeans/Code blocks; all of them fire off a "cmd" or "shell" instance when invoking the link tool rather than call CreateProcess with "g++" like Visual Studio (at least that's what I think based on the information provided). Also, appreciate the patch but I don't think I'll take that route; re-building the toolset is something I would rather not do. Plus, if I'm not mistaken the patch fixes the 'make' side of things, right? In that case, it still wouldn't solve my issue since Eclipse CDT uses an internal builder which doesn't invoke make. BTW, I have already tried the @file option, it doesn't work and fails with the same error. Regarding "group stuff together and incremental linking", unfortunately I'm aware of neither of those techniques. And yes, my last resort will be to install MingW toolchain on my *nix box (maybe in a VM) and compile a DLL but knowing my luck I'm pretty sure that won't go smooth either. :( My next step here would be to try out invoking the "long" command line using my custom tool written in Python/C++ which will use CreateProcess to invoke "g++ -shared". Let's see how that works out! I'll share my findings here. Stephen Lee-13 wrote: > > Sorry, I don't have a fix for your problem, but this bug may be > relevent: > > <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45749> > > Maybe it isn't fixed yet? > I stumbled upon this bug report when trying to find a solution but dismissed it since it mentioned collect2.exe which I wasn't using directly/explicitly? Come to think of it, this may be the reason why my g++ failed even after I passed it the command line arguments via @file. Drats! - sasuke -- View this message in context: http://old.nabble.com/Dealing-with-extremely-large-command-line-when-linking-files-tp34323030p34327640.html Sent from the MinGW - User mailing list archive at Nabble.com. |
|
From: Joe B. <joe...@co...> - 2012-08-21 10:07:12
|
On 21/08/12 07:06, sasuke wrote: > Thanks a lot for your replies folks! I'll try to address each of them below. > > > L.R.N wrote: >> On 20.08.2012 12:26, sasuke wrote: >>> <snip explanation> >>> >> Well, for C i would have used an intermediate library trick: compile >> some .o files into static .a libraries, then link these libraries >> together. Not sure if C++ supports such things. >> > This is what I initially had in mind i.e. merging the .o files created in > batches of 100 and then firing the final command with the newly created 8 .o > files (assuming there are 800 .o files originally). But then I realized that > this method is way too involved and painful to automate. Plus I was really > looking for the reason as to why is this giving me problems. > > > Joe Burmeister wrote: >> Hi Sasuke, >> >> This is similar to what I hit a while ago, patch attached. >> >> Now in my case I couldn't switch to using the @ option file because the >> make files where auto-generated and that would have been a lot more work >> to switch to. >> Instead, I increased the active command line limit, where I was hitting >> it, from 8k to 64k. >> Unless invoke from CreateProcess, the Windows command line only has a 8k >> limit, only this isn't enforced everywhere, causing, strangeness..... >> If the command line is called directly (i.e. not via cmd.exe) from >> CreateProcess, you get the higher 64k limit. >> >> For you this may or may not help as your line may still be too long. It >> would be easy to argue this is Windows fault not MinGW, but that won't >> help you. >> >> >> The way I see it, you can: >> >> * Use the @ option file, >> http://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html >> * Group stuff together into libs and link those (using a def file to >> insure you still get the right DLL interface). >> * Do incremental linking. >> * Use MinGW on Linux to cross build your project. ;-) >> >> Hope this helps. :-) >> > Thanks Joe, I finally see the underlying issue. I really wasn't aware that > there was another way of doing this and was banging my head thinking how was > Visual Studio able to cope up with the large argument size. It's kind of > weird that pretty much *every* free IDE on Windows out there is broken in > this regard. Eclipse CDT/ Netbeans/Code blocks; all of them fire off a "cmd" > or "shell" instance when invoking the link tool rather than call > CreateProcess with "g++" like Visual Studio (at least that's what I think > based on the information provided). Hi Sasuke, What it is, is that most things call the gcc via popen. It's a nice cross platform api call that does exactly what is required, returns a file pipe of the stdout for a command-line, line. The problem is, is that popen's specification is to call via the shell. MS shell is cmd.exe, so MS's popen invokes cmd.exe. So it's stuck on the ye-old 8k command line limit. I was hitting this with windres.exe and the way I got round it was to replace the MS popen with a custom one that use CreateProcess with the command line directly. This meant my windres.exe had the 64k limit and all was good. That's included in the patch. One solution that would be all encompassing is for MinGW to not link against MS's popen but instead a custom one, thus everything cross platform compiled with MinGW would get access to the 64k limit instead of the 8k one. To make use of this across existing apps, you could do a horrible hack. You could hook into the DLL (MSVCRT.DLL) providing MS's _popen and override (with Detours)..... Evil as hell, but might well work. > Also, appreciate the patch but I don't think I'll take that route; > re-building the toolset is something I would rather not do. Plus, if I'm not > mistaken the patch fixes the 'make' side of things, right? In that case, it > still wouldn't solve my issue since Eclipse CDT uses an internal builder > which doesn't invoke make. The other patch was for make, and it is so make uses CreateProcess for a command-line line instead of cmd.exe, regardless if the line contains " or not. For some reason I don't know this was a special character listed that meant the cmd.exe behavior was used. If you aren't using make (you sure Eclipse doesn't?), you gain nothing from this. > BTW, I have already tried the @file option, it doesn't work and fails with > the same error. Regarding "group stuff together and incremental linking", > unfortunately I'm aware of neither of those techniques. And yes, my last > resort will be to install MingW toolchain on my *nix box (maybe in a VM) and > compile a DLL but knowing my luck I'm pretty sure that won't go smooth > either. :( I saw after I pressed send you mentioned the @file already. Sorry about that. I didn't know it was broken! The VisualStudio compiler and linker also seam to have a @file option. The IDE may well have some logic somewhere which is something like, if line short - just do, else - use temp options @file. By grouping stuff together into libs, I just mean, make some static libs for stuff that naturally groups together and build the DLL out of those static libs. You may well need to use a def file to get the DLL interface you want though (the __declspec export isn't preserved). As for incremental linking (also called partial linking) (-i or -r), I've not actually had cause to do it myself, but it means you don't do all the linking at once. My understanding is you do something like: ld -shared -i -o mydll.dll something.o somethingelse.o somethingmore.o ld -shared -i -o mydll.dll yetmore.o andyetmore.o You may well find that cross building on the *nix box goes quite well. I found it all worked rather well and did side step other issues I was having. > My next step here would be to try out invoking the "long" command line using > my custom tool written in Python/C++ which will use CreateProcess to invoke > "g++ -shared". Let's see how that works out! I'll share my findings here. I can't see why that wouldn't work. The only question is, will that be long enough. How long is the line your trying to do? > > > Stephen Lee-13 wrote: >> Sorry, I don't have a fix for your problem, but this bug may be >> relevent: >> >> <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45749> >> >> Maybe it isn't fixed yet? >> > I stumbled upon this bug report when trying to find a solution but dismissed > it since it mentioned collect2.exe which I wasn't using directly/explicitly? > Come to think of it, this may be the reason why my g++ failed even after I > passed it the command line arguments via @file. Drats! Sorry, can't help with that, don't know anything about it. The @file wasn't a direction I explored far as it would have been more work. Hope something in my rambling is helpful to you. Joe |
|
From: LRN <lr...@gm...> - 2012-08-21 06:33:10
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 21.08.2012 10:06, sasuke wrote: > > Thanks a lot for your replies folks! I'll try to address each of > them below. > > > L.R.N wrote: >> >> On 20.08.2012 12:26, sasuke wrote: >>> >>> <snip explanation> >>> >> Well, for C i would have used an intermediate library trick: >> compile some .o files into static .a libraries, then link these >> libraries together. Not sure if C++ supports such things. >> > This is what I initially had in mind i.e. merging the .o files > created in batches of 100 and then firing the final command with > the newly created 8 .o files (assuming there are 800 .o files > originally). But then I realized that this method is way too > involved and painful to automate. Plus I was really looking for the > reason as to why is this giving me problems. Can't you just divide that heap of files into _meaningful_ groups (not just "first 100 in alphabetical order", "second 100...")? That would make the division permanent, you'll have to do it only once. I mean, 800...What the hell have you been trying to do with that insane amount of source files? :) As for the reason: command line length limit is a problem on W32, and, as you can see, the solution is not always straightforward. Though @file feature should have worked. If it doesn't, maybe it's a gcc bug that should be fixed. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJQMyuYAAoJEOs4Jb6SI2CwD5sIALUOFsmmTq1DxROR293Hlo/c 2hDMWrSr3O8vRKnTFCdxr6IoGqrWF+xlT4pTeUUQWSWpJxN3sTO2cBliFxFgPXCP iSk4TCCUUVsG+xocaMP+q5qONOLZ/lDQBlieIIKDwJaHXrLSUbRDT19kjeVsH8n8 ZM2I/KKtadGRs3eUSTO5Quf064KtsvD+4KeFC4jf/aStTX/Tdhhur05GbcZGQukV FlZF+zjzhAmTNJCwcQwhwq0eenCuh5lblgX57w2NR6IXt4g5p3NxzgOr/b/teIoc wfcAKrxiZH4oNYOoiphls0ZL3/9liVnSDVqmumuVZZ/cReGZJwBijnjDjSK+1Ts= =LWj1 -----END PGP SIGNATURE----- |
|
From: sasuke <uzu...@gm...> - 2012-08-25 07:41:59
|
L.R.N wrote: > > On 21.08.2012 10:06, sasuke wrote: >> >> Thanks a lot for your replies folks! I'll try to address each of >> them below. >> >> >> L.R.N wrote: >>> >>> On 20.08.2012 12:26, sasuke wrote: >>>> >>>> <snip explanation> >>>> >>> Well, for C i would have used an intermediate library trick: >>> compile some .o files into static .a libraries, then link these >>> libraries together. Not sure if C++ supports such things. >>> >> This is what I initially had in mind i.e. merging the .o files >> created in batches of 100 and then firing the final command with >> the newly created 8 .o files (assuming there are 800 .o files >> originally). But then I realized that this method is way too >> involved and painful to automate. Plus I was really looking for the >> reason as to why is this giving me problems. > > Can't you just divide that heap of files into _meaningful_ groups (not > just "first 100 in alphabetical order", "second 100...")? That would > make the division permanent, you'll have to do it only once. > I mean, 800...What the hell have you been trying to do with that > insane amount of source files? :) > > As for the reason: command line length limit is a problem on W32, and, > as you can see, the solution is not always straightforward. > > Though @file feature should have worked. If it doesn't, maybe it's a > gcc bug that should be fixed. > Haha, fair enough. I wasn't aware that 800 file single project wasn't a common occurrence. I always thought that almost all big C++ projects out there (open source and not open source) had lots and lots of file and since it was almost always a single project, there has to be a easy solution to my problem. But seems like I was mistaken. So I'm curious, what is a good way of organizing a big C++ project? Are you saying that all big C++ projects are sub-divided into smaller projects? I ask this because I see no way of logical separating the current codebase. :/ Joe Burmeister wrote: > > On 21/08/12 07:06, sasuke wrote: >> Thanks a lot for your replies folks! I'll try to address each of them >> below. >> >> >> L.R.N wrote: >>> On 20.08.2012 12:26, sasuke wrote: >>>> <snip explanation> >>>> >>> Well, for C i would have used an intermediate library trick: compile >>> some .o files into static .a libraries, then link these libraries >>> together. Not sure if C++ supports such things. >>> >> This is what I initially had in mind i.e. merging the .o files created in >> batches of 100 and then firing the final command with the newly created 8 >> .o >> files (assuming there are 800 .o files originally). But then I realized >> that >> this method is way too involved and painful to automate. Plus I was >> really >> looking for the reason as to why is this giving me problems. >> >> >> Joe Burmeister wrote: >>> Hi Sasuke, >>> >>> This is similar to what I hit a while ago, patch attached. >>> >>> Now in my case I couldn't switch to using the @ option file because the >>> make files where auto-generated and that would have been a lot more work >>> to switch to. >>> Instead, I increased the active command line limit, where I was hitting >>> it, from 8k to 64k. >>> Unless invoke from CreateProcess, the Windows command line only has a 8k >>> limit, only this isn't enforced everywhere, causing, strangeness..... >>> If the command line is called directly (i.e. not via cmd.exe) from >>> CreateProcess, you get the higher 64k limit. >>> >>> For you this may or may not help as your line may still be too long. It >>> would be easy to argue this is Windows fault not MinGW, but that won't >>> help you. >>> >>> >>> The way I see it, you can: >>> >>> * Use the @ option file, >>> http://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html >>> * Group stuff together into libs and link those (using a def file to >>> insure you still get the right DLL interface). >>> * Do incremental linking. >>> * Use MinGW on Linux to cross build your project. ;-) >>> >>> Hope this helps. :-) >>> >> Thanks Joe, I finally see the underlying issue. I really wasn't aware >> that >> there was another way of doing this and was banging my head thinking how >> was >> Visual Studio able to cope up with the large argument size. It's kind of >> weird that pretty much *every* free IDE on Windows out there is broken in >> this regard. Eclipse CDT/ Netbeans/Code blocks; all of them fire off a >> "cmd" >> or "shell" instance when invoking the link tool rather than call >> CreateProcess with "g++" like Visual Studio (at least that's what I think >> based on the information provided). > > Hi Sasuke, > > What it is, is that most things call the gcc via popen. It's a nice > cross platform api call that does exactly what is required, returns a > file pipe of the stdout for a command-line, line. > > The problem is, is that popen's specification is to call via the shell. > MS shell is cmd.exe, so MS's popen invokes cmd.exe. So it's stuck on the > ye-old 8k command line limit. > > I was hitting this with windres.exe and the way I got round it was to > replace the MS popen with a custom one that use CreateProcess with the > command line directly. This meant my windres.exe had the 64k limit and > all was good. That's included in the patch. > > One solution that would be all encompassing is for MinGW to not link > against MS's popen but instead a custom one, thus everything cross > platform compiled with MinGW would get access to the 64k limit instead > of the 8k one. > > To make use of this across existing apps, you could do a horrible hack. > You could hook into the DLL (MSVCRT.DLL) providing MS's _popen and > override (with Detours)..... Evil as hell, but might well work. > Hi Joe, thanks for your reply. But are you sure it's 64K? I could have sworn I read 32K on the website and my python script confirmed it. The command line emitted by Eclipse CDT was around 32K and it was still blowing up. I got around it by moving all object files to the same directory and shortening the command even further to 23K. I was then able to build the DLL but now I have a different set of problems. Maybe I'll post it in a separate thread. Joe Burmeister wrote: > >> Also, appreciate the patch but I don't think I'll take that route; >> re-building the toolset is something I would rather not do. Plus, if I'm >> not >> mistaken the patch fixes the 'make' side of things, right? In that case, >> it >> still wouldn't solve my issue since Eclipse CDT uses an internal builder >> which doesn't invoke make. > > The other patch was for make, and it is so make uses CreateProcess for a > command-line line instead of cmd.exe, regardless if the line contains " > or not. For some reason I don't know this was a special character listed > that meant the cmd.exe behavior was used. If you aren't using make (you > sure Eclipse doesn't?), you gain nothing from this. > >> BTW, I have already tried the @file option, it doesn't work and fails >> with >> the same error. Regarding "group stuff together and incremental linking", >> unfortunately I'm aware of neither of those techniques. And yes, my last >> resort will be to install MingW toolchain on my *nix box (maybe in a VM) >> and >> compile a DLL but knowing my luck I'm pretty sure that won't go smooth >> either. :( > > I saw after I pressed send you mentioned the @file already. Sorry about > that. I didn't know it was broken! > The VisualStudio compiler and linker also seam to have a @file option. > The IDE may well have some logic somewhere which is something like, if > line short - just do, else - use temp options @file. > > By grouping stuff together into libs, I just mean, make some static libs > for stuff that naturally groups together and build the DLL out of those > static libs. You may well need to use a def file to get the DLL > interface you want though (the __declspec export isn't preserved). > > As for incremental linking (also called partial linking) (-i or -r), > I've not actually had cause to do it myself, but it means you don't do > all the linking at once. > My understanding is you do something like: > > ld -shared -i -o mydll.dll something.o somethingelse.o somethingmore.o > ld -shared -i -o mydll.dll yetmore.o andyetmore.o > > > You may well find that cross building on the *nix box goes quite well. I > found it all worked rather well and did side step other issues I was > having. > >> My next step here would be to try out invoking the "long" command line >> using >> my custom tool written in Python/C++ which will use CreateProcess to >> invoke >> "g++ -shared". Let's see how that works out! I'll share my findings here. > > I can't see why that wouldn't work. The only question is, will that be > long enough. How long is the line your trying to do? > As mentioned above, it was around 32K with Eclipse (and a whopping 53K with Code blocks due to long paths) and it was failing. I solved it as mentioned above. >> Stephen Lee-13 wrote: >>> Sorry, I don't have a fix for your problem, but this bug may be >>> relevent: >>> >>> <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45749> >>> >>> Maybe it isn't fixed yet? >>> >> I stumbled upon this bug report when trying to find a solution but >> dismissed >> it since it mentioned collect2.exe which I wasn't using >> directly/explicitly? >> Come to think of it, this may be the reason why my g++ failed even after >> I >> passed it the command line arguments via @file. Drats! > > Sorry, can't help with that, don't know anything about it. The @file > wasn't a direction I explored far as it would have been more work. > > Hope something in my rambling is helpful to you. > Thanks for the food for thought; I have slowly come to appreciate the intricacies involved in fighting against the tide called MSVC toolchain; but like they say "the worst if yet to come". :) /sasuke -- View this message in context: http://old.nabble.com/Dealing-with-extremely-large-command-line-when-linking-files-tp34323030p34347155.html Sent from the MinGW - User mailing list archive at Nabble.com. |
|
From: Gisle V. <gv...@br...> - 2012-08-25 12:01:13
|
"sasuke" <uzu...@gm...> wrote: > Haha, fair enough. I wasn't aware that 800 file single project wasn't a > common occurrence. I always thought that almost all big C++ projects out > there (open source and not open source) had lots and lots of file and since > it was almost always a single project, there has to be a easy solution to my > problem. But seems like I was mistaken. I've been playing with (and building) Qt-SDK [1] lately. This uses the method below of dealing with long list of .o-lists for the linker. The (generated) makefiles has a link rule something like this: foo.dll: $(OBJECTS) object_list.tmp $(CC) -o $@ -shared -Wl,--out-implib,libfoo.dll.a $(OBJECTS) object_list.tmp object_list.tmp contain things like: INPUT( ./tmp/obj/release_shared/qabstractanimation.o ./tmp/obj/release_shared/qvariantanimation.o ./tmp/obj/release_shared/qpropertyanimation.o <etc.> ); which you can create with 'echo' + 'ls -l' etc. So the problem and solution isn't new. [1] http://qt.nokia.com/ --gv |
|
From: Joe B. <joe...@co...> - 2012-08-28 08:33:53
|
Hi Sasuke > Hi Joe, thanks for your reply. But are you sure it's 64K? I could have sworn > I read 32K on the website and my python script confirmed it. The command > line emitted by Eclipse CDT was around 32K and it was still blowing up. I > got around it by moving all object files to the same directory and > shortening the command even further to 23K. I was then able to build the DLL > but now I have a different set of problems. Maybe I'll post it in a separate > thread. > Looks like you are right: http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx My memory is clearly playing tricks on me. I written it as 64k to some others too.... hope they work that mistake out for themselves! > > As mentioned above, it was around 32K with Eclipse (and a whopping 53K with > Code blocks due to long paths) and it was failing. I solved it as mentioned > above. > Cool, glad this one is solved for you. :-) Joe |
|
From: Tomi O. <tom...@ik...> - 2012-08-28 13:27:13
|
On Tue, Aug 28 2012, Joe Burmeister wrote: > Hi Sasuke > >> Hi Joe, thanks for your reply. But are you sure it's 64K? I could have sworn >> I read 32K on the website and my python script confirmed it. The command >> line emitted by Eclipse CDT was around 32K and it was still blowing up. I >> got around it by moving all object files to the same directory and >> shortening the command even further to 23K. I was then able to build the DLL >> but now I have a different set of problems. Maybe I'll post it in a separate >> thread. >> > > Looks like you are right: > > http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx > > My memory is clearly playing tricks on me. I written it as 64k to some > others too.... hope they work that mistake out for themselves! Something related; IIRC something like perl -le 'exec "echo", (20) x 1e5' works on MSYS command line; that executes echo with 100,000 arguments, every arg 2 characters long. This must do it in some other way than concatenating string, as this would make the arg 300,000 octets in length. I've used this construct in perl program executing 'md5sum' for a long list of files -- MSYS md5sum to be precise -- what happens when executing non-MSYS program might be different. I cannot test now as I don't have any windows machines around... so if you're interested test on your MSYS shell. >> As mentioned above, it was around 32K with Eclipse (and a whopping 53K with >> Code blocks due to long paths) and it was failing. I solved it as mentioned >> above. >> > > Cool, glad this one is solved for you. :-) > > Joe Tomi |
|
From: Joe B. <joe...@co...> - 2012-08-28 14:49:08
|
On 28/08/12 13:54, Tomi Ollila wrote: > On Tue, Aug 28 2012, Joe Burmeister wrote: > >> Hi Sasuke >> >>> Hi Joe, thanks for your reply. But are you sure it's 64K? I could have sworn >>> I read 32K on the website and my python script confirmed it. The command >>> line emitted by Eclipse CDT was around 32K and it was still blowing up. I >>> got around it by moving all object files to the same directory and >>> shortening the command even further to 23K. I was then able to build the DLL >>> but now I have a different set of problems. Maybe I'll post it in a separate >>> thread. >>> >> >> Looks like you are right: >> >> http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx >> >> My memory is clearly playing tricks on me. I written it as 64k to some >> others too.... hope they work that mistake out for themselves! > > > Something related; IIRC something like > > perl -le 'exec "echo", (20) x 1e5' > > works on MSYS command line; that executes echo with 100,000 arguments, > every arg 2 characters long. > > This must do it in some other way than concatenating string, as this would > make the arg 300,000 octets in length. > > I've used this construct in perl program executing 'md5sum' for a long list > of files -- MSYS md5sum to be precise -- what happens when executing > non-MSYS program might be different. > > > I cannot test now as I don't have any windows machines around... so if > you're interested test on your MSYS shell. > I've left this for now as I fixed it in mingw32-make and windres for me so I could complete the project in hand and gave the patches to the client. The spec was to use the Windows command line with the MinGW tools. I was pleased to get away from Windows back to Linux to be honest. No doubt I'll be on some Windows MinGW project again at some point though. I might look then if I hit this then. Joe |
|
From: sasuke <uzu...@gm...> - 2012-09-01 08:08:53
|
Tomi Ollila wrote: > > On Tue, Aug 28 2012, Joe Burmeister wrote: > >> Hi Sasuke >> >>> Hi Joe, thanks for your reply. But are you sure it's 64K? I could have >>> sworn >>> I read 32K on the website and my python script confirmed it. The command >>> line emitted by Eclipse CDT was around 32K and it was still blowing up. >>> I >>> got around it by moving all object files to the same directory and >>> shortening the command even further to 23K. I was then able to build the >>> DLL >>> but now I have a different set of problems. Maybe I'll post it in a >>> separate >>> thread. >>> >> >> Looks like you are right: >> >> http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx >> >> My memory is clearly playing tricks on me. I written it as 64k to some >> others too.... hope they work that mistake out for themselves! > > > Something related; IIRC something like > > perl -le 'exec "echo", (20) x 1e5' > > works on MSYS command line; that executes echo with 100,000 arguments, > every arg 2 characters long. > > This must do it in some other way than concatenating string, as this would > make the arg 300,000 octets in length. > > I've used this construct in perl program executing 'md5sum' for a long > list > of files -- MSYS md5sum to be precise -- what happens when executing > non-MSYS program might be different. > > > I cannot test now as I don't have any windows machines around... so if > you're interested test on your MSYS shell. > > Hi Tomi, that seemed like a nice trick but alas here is the output on my MSYS: C:\mingw\msys\1.0\bin>perl -le 'exec "echo", (20) x 1e5' C:\mingw\msys\1.0\bin\perl.exe: *** couldn't commit memory for cygwin heap, Win32 error 487 Maybe you were trying it on a 64-bit Windows OS? /sasuke -- View this message in context: http://old.nabble.com/Dealing-with-extremely-large-command-line-when-linking-files-tp34323030p34376637.html Sent from the MinGW - User mailing list archive at Nabble.com. |
|
From: Earnie B. <ea...@us...> - 2012-09-01 16:08:41
|
On Sat, Sep 1, 2012 at 4:08 AM, sasuke <uzu...@gm...> wrote: > > > Tomi Ollila wrote: >> >> On Tue, Aug 28 2012, Joe Burmeister wrote: >> >>> Hi Sasuke >>> >>>> Hi Joe, thanks for your reply. But are you sure it's 64K? I could have >>>> sworn >>>> I read 32K on the website and my python script confirmed it. The command >>>> line emitted by Eclipse CDT was around 32K and it was still blowing up. >>>> I >>>> got around it by moving all object files to the same directory and >>>> shortening the command even further to 23K. I was then able to build the >>>> DLL >>>> but now I have a different set of problems. Maybe I'll post it in a >>>> separate >>>> thread. >>>> >>> >>> Looks like you are right: >>> >>> http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx >>> >>> My memory is clearly playing tricks on me. I written it as 64k to some >>> others too.... hope they work that mistake out for themselves! >> >> >> Something related; IIRC something like >> >> perl -le 'exec "echo", (20) x 1e5' >> >> works on MSYS command line; that executes echo with 100,000 arguments, >> every arg 2 characters long. >> >> This must do it in some other way than concatenating string, as this would >> make the arg 300,000 octets in length. >> >> I've used this construct in perl program executing 'md5sum' for a long >> list >> of files -- MSYS md5sum to be precise -- what happens when executing >> non-MSYS program might be different. >> >> >> I cannot test now as I don't have any windows machines around... so if >> you're interested test on your MSYS shell. >> >> > Hi Tomi, that seemed like a nice trick but alas here is the output on my > MSYS: > > C:\mingw\msys\1.0\bin>perl -le 'exec "echo", (20) x 1e5' > C:\mingw\msys\1.0\bin\perl.exe: *** couldn't commit memory for cygwin heap, > Win32 error 487 > > Maybe you were trying it on a 64-bit Windows OS? No, not the reason. I executed this on both an old crusty XP Win32 laptop and my new Win64 laptop without receiving this error. Maybe you have some BLODA issue? -- Earnie -- https://sites.google.com/site/earnieboyd |
|
From: SL <sld...@so...> - 2012-08-20 09:24:53
|
sasuke <uzu...@gm...> wrote: > > I have tried pretty much every trick I knew of. Using the @file flag for g++ > etc. but it still doesn't work. Sorry, I don't have a fix for your problem, but this bug may be relevent: <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45749> Maybe it isn't fixed yet? -- SL <sld...@so...> |
|
From: Joe B. <joe...@co...> - 2012-08-20 09:26:04
Attachments:
cmd_limit.tar.bz2
|
Hi Sasuke, This is similar to what I hit a while ago, patch attached. Now in my case I couldn't switch to using the @ option file because the make files where auto-generated and that would have been a lot more work to switch to. Instead, I increased the active command line limit, where I was hitting it, from 8k to 64k. Unless invoke from CreateProcess, the Windows command line only has a 8k limit, only this isn't enforced everywhere, causing, strangeness..... If the command line is called directly (i.e. not via cmd.exe) from CreateProcess, you get the higher 64k limit. For you this may or may not help as your line may still be too long. It would be easy to argue this is Windows fault not MinGW, but that won't help you. The way I see it, you can: * Use the @ option file, http://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html * Group stuff together into libs and link those (using a def file to insure you still get the right DLL interface). * Do incremental linking. * Use MinGW on Linux to cross build your project. ;-) Hope this helps. :-) Joe On 20/08/12 09:26, sasuke wrote: > Hi folks, > > I have been trying to "build" a shared-dynamic library (DLL) on Windows for > a largish code base (around 800 CPP files which result in 800 .o files) > using the MingW toolset and seem to have hit upon a snag. The problem occurs > during the linking phase when a single large command is created which has > *all* the .o files on it. Something like: > > g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll > > I have tried out pretty much every free IDE out there on Windows and the > result is the same. The object files are created all right but the linking > step fails every time with some weird error message which ultimately boils > down to "too long argument list". For e.g. on Netbeans, the error I get is > "make[2]: execvp: g++: Bad file number" which as per my searching again > comes back to the long list of object files. Strangely enough, Visual studio > seems to work fine with the given project and successfully creates the DLL > even with the long command line. > > I have tried pretty much every trick I knew of. Using the @file flag for g++ > etc. but it still doesn't work. Is there *anyone* out there who has > successfully managed to create a DLL for a largish project using the MingW > toolset. Please let me know if there are any queries or additional > information needed. Any assistance/tips would be greatly appreciated! > > TIA, > sasuke |
|
From: Earnie B. <ea...@us...> - 2012-08-20 13:29:31
|
On Mon, Aug 20, 2012 at 4:26 AM, sasuke wrote: > > Hi folks, > > I have been trying to "build" a shared-dynamic library (DLL) on Windows for > a largish code base (around 800 CPP files which result in 800 .o files) > using the MingW toolset and seem to have hit upon a snag. The problem occurs > during the linking phase when a single large command is created which has > *all* the .o files on it. Something like: > > g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll > One option that others did not mention is that you can create one object file from many C/C++ source files and effectively reduce the number of objects files to link with. cat a.c b.c c.c ... etc | gcc -o a.o -xc - -- Earnie -- https://sites.google.com/site/earnieboyd |
|
From: John B. <joh...@ho...> - 2012-08-20 17:46:15
|
On Mon, 20 Aug 2012 09:29:24 -0400, Earnie Boyd wrote:
>
> On Mon, Aug 20, 2012 at 4:26 AM, sasuke wrote:
> >
> > Hi folks,
> >
> > I have been trying to "build" a shared-dynamic library (DLL) on Windows for
> > a largish code base (around 800 CPP files which result in 800 .o files)
> > using the MingW toolset and seem to have hit upon a snag. The problem occurs
> > during the linking phase when a single large command is created which has
> > *all* the .o files on it. Something like:
> >
> > g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll
> >
>
> One option that others did not mention is that you can create one
> object file from many C/C++ source files and effectively reduce the
> number of objects files to link with.
>
> cat a.c b.c c.c ... etc | gcc -o a.o -xc -
>
> --
> Earnie
That won't work for something like:
/* test1.c */
/* f is visible in this translation unit only */
static void f(){}
/* end test1.c */
/* test2.c */
/* f is visible in all translation units */
void f(){}
/* end test2.c */
/* test3.c */
int main()
{
f();
return 0;
}
/* end test3.c */
$ cat test1.c test2.c test3.c | gcc -o test.o -xc -
<stdin>:10:6: error: redefinition of 'f'
<stdin>:4:13: note: previous definition of 'f' was here
Besides, doesn't this method defeat the purpose of make?
You are recompiling source files whether they have changed
or not.
Regards,
John Brown.
|
|
From: Earnie B. <ea...@us...> - 2012-08-20 18:49:58
|
On Mon, Aug 20, 2012 at 1:46 PM, John Brown wrote:
>
> On Mon, 20 Aug 2012 09:29:24 -0400, Earnie Boyd wrote:
>>
>> On Mon, Aug 20, 2012 at 4:26 AM, sasuke wrote:
>> >
>> > Hi folks,
>> >
>> > I have been trying to "build" a shared-dynamic library (DLL) on Windows for
>> > a largish code base (around 800 CPP files which result in 800 .o files)
>> > using the MingW toolset and seem to have hit upon a snag. The problem occurs
>> > during the linking phase when a single large command is created which has
>> > *all* the .o files on it. Something like:
>> >
>> > g++.exe -shared a.o b.o c.o .... zzz.o -o some.dll
>> >
>>
>> One option that others did not mention is that you can create one
>> object file from many C/C++ source files and effectively reduce the
>> number of objects files to link with.
>>
>> cat a.c b.c c.c ... etc | gcc -o a.o -xc -
>>
>> --
>> Earnie
>
>
> That won't work for something like:
>
> /* test1.c */
>
> /* f is visible in this translation unit only */
> static void f(){}
>
> /* end test1.c */
>
> /* test2.c */
>
>
>
> /* f is visible in all translation units */
>
> void f(){}
>
>
>
> /* end test2.c */
>
> /* test3.c */
> int main()
> {
> f();
> return 0;
> }
>
> /* end test3.c */
>
> $ cat test1.c test2.c test3.c | gcc -o test.o -xc -
> <stdin>:10:6: error: redefinition of 'f'
> <stdin>:4:13: note: previous definition of 'f' was here
>
Yea, I wasn't thinking too much about it.
> Besides, doesn't this method defeat the purpose of make?
> You are recompiling source files whether they have changed
> or not.
Depends on how you setup the target and dependencies doesn't it?
test.o: test1.c test2.c test3.c
cat test1.c test2.c test3.c | gcc -o test.o -xc -
The test.o will only be recompiled if test1.c or test2.c or test3.c is
newer but I suppose you mean that if one changes they all need to be
recompiled. Command line length has always been an issue for large
number of files and this is a potential method for dealing with it but
only if there is no collision (as you point out) in the source. The
better method will be to create a static library that is used to
create the shared library. You can break out the object files into
smaller chunks adding the pieces to the end of the static library.
--
Earnie
-- https://sites.google.com/site/earnieboyd
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X