Bugs item #3072100, was opened at 2010-09-20 19:13
Message generated for change (Comment added) made by keithmarshall
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=3072100&group_id=2435
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: MSYS
Group: Known Feature
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Bob Friesenhahn (bfriesen)
Assigned to: Nobody/Anonymous (nobody)
Summary: MSYS make is 60X slower than before
Initial Comment:
MSYS make 3.82 is now 60X slower than earlier make programs. When building GraphicsMagick, it takes 2.5 minutes before make does any work or decides there is nothing to be done. After make has decided what to do, the build proceeds normally. It normally takes a couple of seconds before make decides what needs to be done. With a Cygwin installation on the same machine using the same source files and same build parameters, make decides there is nothing to be done in about two seconds. The build is done with the source files accessed via CIFS to a fast server and object files written locally. Using 'make -d' shows that there are about 340K rules which are tested, without any noticeable pauses or slowness due to server access.
Charles Wilson tells me that make now tests various permutations of upper and lower case in order to handle filesystems which don't preserve case. This approach would certainly dramatically multiply the number of filesystem acceses and may explain the dramatic slowdown. I also notice that make 3.82 tests about twice as many implicit rules than make 3.81 does.
----------------------------------------------------------------------
>Comment By: Keith Marshall (keithmarshall)
Date: 2011-04-26 11:57
Message:
case preserving != case sensitive
The windows file system is case insensitive, pretending otherwise can have
serious, (and for all practical purposes, insoluble), consequences.
Pattern rules such as
%.$(OBJEXT): %.c
will fail dramatically, if the compiler doesn't strictly honour the case
preserving concept; even for an 8.3 file name, the number of permutations
of this rule which must be written, if you try to pretend that
case preserving == case sensitive
becomes prohibitive. YOU may not have encountered this issue, but I have;
indeed, it is the fundamental reason why cygwin was rejected out of hand,
for porting a legacy DJGPP application, in my day job; (cygwin's make
promotes the case preserving == case sensitive fiction, as did early MSYS
builds).
Granted, problems are more likely to arise with legacy MS-DOS
applications, than with anything developed from scratch, for more modern
WinNT based platforms. When I first raised the issue, several years ago,
it was a serious problem for me; the solution was to ensure that make was
built with its CASE_INSENSITIVE_FILESYSTEM option enabled. It would be
less of an issue for me today, because I no longer have to maintain that
legacy application. However I still believe it is important that we
continue to offer a make which does respect the true case insensitive
nature of the file system.
When we first addressed the issue, we offered two variants of make; the
default was built with CASE_INSENSITIVE_FILESYSTEM, (which I still believe
to be the safest option), and the alternative called csmake was offered for
those who wished to promulgate the case preserving == case sensitive
fiction.
Later, Cesar devised the current hybrid build, which attempts to be all
things to all men. It seemed like a good idea, at the time. Maybe the
speed issue is just too serious a detraction, and we should revert to the
two alternative choices, leaving users to decide which is most suitable for
them to install as make.exe
----------------------------------------------------------------------
Comment By: Earnie Boyd (earnie)
Date: 2011-04-25 17:26
Message:
Case preserving case insensitive file name operation is the issue. Yes, it
preserves case but files named FOO, Foo, fOO, fOo and foo will all be found
regardless of the reference. The patch allows for non-matching targets to
be considered so that the target FOO doesn't build if foo is found and is
up-to-date. Frankly, I have never had an issue with make without the
patch.
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2011-04-25 14:22
Message:
Since Windows is case-preserving, I tend to wonder about the decision for
'make' to be case insensitive by default. It seems like a bad idea to me
since those developing projects primarily under MinGW but with a desire to
be portable to other systems (e.g. Linux, or even Cygwin) will potentially
produce projects which don't work at all due to case issues.
My project (GraphicsMagick) is open sourced and soon I will be encouraging
users to build it from source on MinGW rather than MSVC. If 'make' seems
incredibly slow, I will soon hear complaints.
Regardless, I did already add a shell alias in order to improve my
personal efficiency.
----------------------------------------------------------------------
Comment By: Earnie Boyd (earnie)
Date: 2011-04-25 11:39
Message:
I was just trying to eliminate possibilities. I agree with you, IIRC
csmake was added as a result of conversations I had made on one of the
lists (mingw-users or mingw-dvlpr). Solution to your problem is to set in
your ~/.profile file ``alias make=csmake'' and let's be done with the
conversation. I would encourage you though to create a wiki post of your
findings and the solution to the issue I've given you.
Keith and Cesar, do you think we should close as won't fix or do you think
we should reverse the decision to include the case insensitive patch as the
default make for MSYS and instead of csmake have one that is named cimake?
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2011-04-23 17:05
Message:
Earnie, I relish data caching, and quite rarely reboot computers. Caching
is good. My timings are done with anything that can be cached already
cached on the client system The file server has 20GB of RAM so it also
does lots of caching. Since the software builds fine using 'csmake' there
is no need for any case-insensitive code to be used.
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2011-04-23 16:57
Message:
To put things in perspective, here is the total package build time from
'csmake -j 4' on the same system:
real 0m46.391s
user 1m15.990s
sys 0m41.554s
With csmake, the whole package could build from scratch in the time that
it takes for 'make' to visibly start doing anything at all.
----------------------------------------------------------------------
Comment By: Earnie Boyd (earnie)
Date: 2011-04-23 16:54
Message:
To eliminate the possibility of data caching affecting the timings you need
to reboot your computer and enter csmake command first.
$ time ls /bin
real 0m0.469s
user 0m0.030s
sys 0m0.154s
$ time ls /bin
real 0m0.391s
user 0m0.030s
sys 0m0.093s
As you can see the second occurrence uses much less system time. Although
I doubt you'll see a big improvement over 2m for make.
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2011-04-23 16:41
Message:
I should mention that GraphicsMagick uses a non-recursive Automake build
configuration so that source file name specifications including a directory
component when building. The source files reside on a fast Samba server
but the build directory is hosted by the Windows system.
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2011-04-23 16:38
Message:
I finally got back to this since today I am using MinGW and it is driving
me crazy. The 'csmake' program runs vastly faster than the standard MinGW
'make' to decide to do nothing for a GraphicsMagick build:
bfriesen@... ~/mingw/GM-16-static
$ time make
make all-am
make[1]: Entering directory `/home/bfriesen/mingw/GM-16-static'
make[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
real 2m14.689s
user 0m5.295s
sys 0m40.890s
bfriesen@... ~/mingw/GM-16-static
$ time csmake
csmake all-am
csmake[1]: Entering directory `/home/bfriesen/mingw/GM-16-static'
csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
real 0m2.562s
user 0m0.858s
sys 0m0.921s
----------------------------------------------------------------------
Comment By: Cesar Strauss (cstrauss)
Date: 2010-09-22 23:04
Message:
Please try the following version of make (csmake) which does not have the
case preserving patch:
http://prdownloads.sourceforge.net/mingw/csmake-3.81-MSYS-1.0.11-2.tar.bz2?download
Does csmake run at the normal speed?
Regards,
Cesar
----------------------------------------------------------------------
Comment By: Bob Friesenhahn (bfriesen)
Date: 2010-09-21 15:01
Message:
My error. It is indeed 3.81:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i686-pc-msys
----------------------------------------------------------------------
Comment By: Cesar Strauss (cstrauss)
Date: 2010-09-21 00:30
Message:
Dear Bob,
Could you send the output of make --version, please?
I'm a bit confused, since the latest MSYS version available is still 3.81.
See
https://sourceforge.net/projects/mingw/files_beta/MSYS/make/
Regards,
Cesar
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=3072100&group_id=2435
|
