Thread: Re: [Libusb-devel] CRLF in files checked out on Windows (Page 8)

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Correcting a missing negation there:

> then, as far as I know, it is
> not possible to use "text" ever and *NOT* end up with LF files in it  (the repo)

/Pete

>>>>> "text" for git means "do whatever line ending conversions you asked
>>>>> for", etc.  It's the default on all files.  It does not mean "LF".
>>>>
>>>> [...] I still don't get how
>>>> "text" does not equate "LF"
>>
>> I've seen both assumptions throughout this thread, which is why I
>> pointed
>> this out.  Segher's response is on the same page with me.
>
> I'm considering text from the point of view of the repo, so maybe that's
> my mistake.

A "text" file, from the user's point of view, is anything that is, well,
text: documentation files, C source files, build system files, etc.

On a unix system, all those have LF line endings.  In a git repo, used
for cross-platform development, they do as well.

In mswindows environments that use LF line endings (cygwin, msys, maybe
all mingw), you use autocrlf=off (or autocrlf=input), and all is fine.

In mswindows environments that use CRLF line endings ("native" mswindows),
you use aurocrlf=true, and all is fine.  (It would seem things will work
for MSVC even with autocrlf=off, but that as an aside).

Yes, mswindows developers do have to consciously decide what to set
autocrlf to.  This can be seen as a shortcoming of git; it is nothing
we can do anything about.  People just have to learn to use their tools.
Any kludges you can come up with to make things "magically work" will
in the end just magically explode in your face (and worse, in the face
of people who have nothing to do with the "problem" in the first place!)

Segher

Segher Boessenkool wrote:

> In mswindows environments that use CRLF line endings ("native" mswindows),
> you use aurocrlf=true, and all is fine.  (It would seem things will work
> for MSVC even with autocrlf=off, but that as an aside).

Note that here is little in a "native" mswindows environment that requires CRLF.
I certainly use MSVC and have no trouble using strictly LF, because
I'm not using notepad, and not editing/using VC project files.
Even MSWin system files such as .inf work fine with LF.
(My source packaging script converts every non-binary file to
  LF line endings, and the result happily compiles cross platform).

Graeme Gill.

On 2010.12.07 16:43, Segher Boessenkool wrote:
> Yes, mswindows developers do have to consciously decide what to set
> autocrlf to.  This can be seen as a shortcoming of git; it is nothing
> we can do anything about.

I think I proved otherwise. With my current branch, nobody has to 
consciously decide how they should set autocrlf. It only matters if they 
want to push patches back to us (and then we can tell them what to use - 
this is what happened to me when I started pushing patches), which only 
concerns a very limited subset of libusb users.

> People just have to learn to use their tools.

Agreed. But we can also avoid them learning more than they should. Not 
everybody wants to be an expert on git CRLF conversion, or tax returns, 
or whatever you are being constrained to learn because you're not given 
any other choice. I don't think we'll ever avoid the need to learn tax 
returns, but we can spare the requirement of learning git CRLF 
conversion from top to bottom where a quick "if you're on Windows and 
want to submit patches to us, use autocrlf. And if you patch 
.ac/.am/.sh, please keep the files LF'd - but git will tell us if you 
introduce CRs there anyway)

> Any kludges you can come up with to make things "magically work" will
> in the end just magically explode in your face (and worse, in the face
> of people who have nothing to do with the "problem" in the first place!)

And so we've come to this. "I don't like your *solution*, and I cannot 
find a default with it on logical grounds, so I'll just call it a 
'kludge' and dangle the idea of *potential* *future* problems that I am 
unable to elaborate on, and that, had I analysed the solution properly, 
would know are not going to happen"

The only potential issue is mixing CRLF in LF files, which we can spot 
and/or fix, and which is not going to make thing "magically explode" in 
anybody's face.

If you want to reject the solution, that's fine. That's your executive 
right. But don't pretend that you are rejecting it on technical grounds, 
or do provide a potential scenario where you expect using -crlf on 
.sh/.ac/.am will effectively break things in the future.

Until then, I'll take the *proven* 3 different scenarios that I 
experienced of non -crlf breaking things for LF files (one for each of 
.sh, .ac, .am, which, since I have seen failure to convert on eol=lf I 
am entitled to assume would have happened with eol=lf as well) over your 
"future blow up" any time.

Why should my 3 *actual* "blow ups" have less importance than your 
future potential unproven "blow up"?

Regards,

/Pete

>> Yes, mswindows developers do have to consciously decide what to set
>> autocrlf to.  This can be seen as a shortcoming of git; it is nothing
>> we can do anything about.
>
> I think I proved otherwise.

I think you merely showed that yes, it is possible to mess things up.

>> People just have to learn to use their tools.
>
> Agreed. But we can also avoid them learning more than they should. Not
> everybody wants to be an expert on git CRLF conversion, or tax returns,
> or whatever you are being constrained to learn because you're not given
> any other choice. I don't think we'll ever avoid the need to learn tax
> returns, but we can spare the requirement of learning git CRLF
> conversion from top to bottom where a quick "if you're on Windows and
> want to submit patches to us, use autocrlf. And if you patch
> .ac/.am/.sh, please keep the files LF'd - but git will tell us if you
> introduce CRs there anyway)

And how is it easier to do special stuff to some files in our repo, which
might or might not hide from the mswindows that they should have set
autocrlf correctly, oh but you need to do this and that in these few cases;
how is that easier then to tell them to set autocrlf *like they need to
do for anything else they use git for*?

"Plain" git already defaults to the correct setting, since that is the
same for everyone; and msysgit *asks* you on install what to default to.

>> Any kludges you can come up with to make things "magically work" will
>> in the end just magically explode in your face (and worse, in the face
>> of people who have nothing to do with the "problem" in the first place!)
>
> And so we've come to this. "I don't like your *solution*, and I cannot
> find a default with it on logical grounds, so I'll just call it a
> 'kludge' and dangle the idea of *potential* *future* problems that I am
> unable to elaborate on, and that, had I analysed the solution properly,
> would know are not going to happen"

I am saying you have a non-solution to a non-problem.  Any problems you
have here are problems you created yourself; and the "solution" is not
a full solution, and it *does* give maintenance headaches (what other
files will we need to set what attrs on later?), and it *might* cause
other problems (if nothing else, as you yourself have noted, this area
is in flux in git; if your argument is "well it's not likely it will
cause problems", I might even agree after thinking on it, but why would
we even consider it).

> Until then, I'll take the *proven* 3 different scenarios that I
> experienced of non -crlf breaking things for LF files

You proved that you managed to break things.  If you can show that
you have problems with sane environments, with sane workflows (that
does not include copying files between LF and CRLF environments!),
then of course we should fix something.

> Why should my 3 *actual* "blow ups" have less importance than your
> future potential unproven "blow up"?

Because _you_ caused your own three problems, and _you_ would cause those
future problems.

Segher

>> "text" for git means "do whatever line ending conversions you asked
>> for", etc.  It's the default on all files.  It does not mean "LF".
>
> So does that mean you can have "text" that is CRLF in the repo?

Yes; for example, someone who uses CRLF and has autocrlf=off.

Such a repo is only usable by mswindows users.

Segher

On 2010.12.07 17:00, Segher Boessenkool wrote:
>> So does that mean you can have "text" that is CRLF in the repo?
>
> Yes; for example, someone who uses CRLF and has autocrlf=off.
>
> Such a repo is only usable by mswindows users.

Interesting. I thought that git would always convert text to LF, but it 
looks like I was wrong. Then we could get the MS project files in the 
repo with CRLF using autocrlf=off (but I wouldn't advocate for this).

I'm not seeing any incidence of text not always equating LF on my 
proposed solution for the .ac/.am/.sh problem though, unless I'm missing 
something.

Regards,

/Pete

>>> So does that mean you can have "text" that is CRLF in the repo?
>>
>> Yes; for example, someone who uses CRLF and has autocrlf=off.
>>
>> Such a repo is only usable by mswindows users.
>
> Interesting. I thought that git would always convert text to LF, but it
> looks like I was wrong. Then we could get the MS project files in the
> repo with CRLF using autocrlf=off (but I wouldn't advocate for this).

They can be marked as binary, or as -crlf, or simply as nothing: everyone
who checks those files in has the same crlf config as everyone who uses
them.  The first two options sound safest, yeah.

> I'm not seeing any incidence of text not always equating LF on my
> proposed solution for the .ac/.am/.sh problem though, unless I'm missing
> something.

What you are missing is that all people who need the build files
with LF line endings, need *all* text files with LF line endings.

Segher

On 2010.12.07 18:16, Segher Boessenkool wrote:
> What you are missing is that all people who need the build files
> with LF line endings, need *all* text files with LF line endings.

Which means that people using Tgit and MSVC (CRLF) are supposed to 
retrieve all their files including the autotool ones with CRLF, and 
cannot switch environments without a new pull (provided they set their 
options right).

Or people who use MinGW/cygwin must ensure they use LF always, even as 
the expectation is that gcc handles LF or CRLF sources transparently, so 
the only files that actually *need* to be LF are non .c/.h

Do we expect to support a cygwin/MinGW gcc that can't handle CRLF? If 
not, then your *all* is inaccurate, and we have no need to enforce LF 
for platforms that are supposed to need LF, apart from the files that 
are not .c/.h. Let's see what these files should be for cygwin or MinGW 
(which are the Windows platforms supposed to require LF):

First we eliminate all the MSVC/DDK stuff. Then, I wouldn't care too 
much about README/TODO etc. The .rc/.def are probably fine either way 
(if not, they would be expected to be CRLF rather than LF, since these 
are supposed to be MS generated files). Which leaves... .sh/.ac/.am as 
the ONLY files that actually *need* LF line endings...

Again, why force-convert files if we don't have to?

Unless we want to consider that MinGW and cygwin's gcc will break on 
CRLF (for a gcc specifically aimed at a platform where the default is 
CRLF), I still don't see much of an issue with the -crlf solution, 
especially as it allows moving files around between envs almost freely

Regards,

/Pete

>> What you are missing is that all people who need the build files
>> with LF line endings, need *all* text files with LF line endings.
>
> Which means that people using Tgit and MSVC (CRLF) are supposed to
> retrieve all their files including the autotool ones with CRLF, and
> cannot switch environments without a new pull (provided they set their
> options right).

Without a new checkout, yes.  A separate tree would certainly help in
keeping the user sane.

> Or people who use MinGW/cygwin must ensure they use LF always,

Yes, they should.  Nothing new there.

> even as
> the expectation is that gcc handles LF or CRLF sources transparently, so
> the only files that actually *need* to be LF are non .c/.h
>
> Do we expect to support a cygwin/MinGW gcc that can't handle CRLF?

Why does that matter?  GCC might handle CRLF line endings the way you
want; many other tools you can use with C files (grep, for one) don't.

> Again, why force-convert files if we don't have to?

Why play fast-and-loose with it if you don't have to?

Segher

On 2010.12.07 19:10, Segher Boessenkool wrote:
>> Do we expect to support a cygwin/MinGW gcc that can't handle CRLF?
>
> Why does that matter?  GCC might handle CRLF line endings the way you
> want; many other tools you can use with C files (grep, for one) don't.

Mmmm, I hadn't really considered that, even though I seem to recall that 
Michael mentioned something like that. That's actually a good point. If 
we expect people to use tools that can't handle CRLF in cygwin and 
MinGW, then yes, I have to review my position.

I think there's probably a valid scenario for wanting to use grep on our 
sources in MinGW/cygwin. I'm actually using sed on our sources in 
autogen to generate the .def so I should have thought about that one 
(though I have not seen any issue with sed).

>> Again, why force-convert files if we don't have to?
>
> Why play fast-and-loose with it if you don't have to?

Because it allows for "insane" ways of speeding up development. ;)

Regards,

/Pete

Pete Batard wrote:
> Just tested grep, more (less), vim, and a few other command line tools 
> in cygwin and MinGW, and at least for the versions I have, they are CRLF 
> aware.

Very nice.

> They will produce LF'd output however, so if we were to use grep to 
> create a source, we'd get CRLF -> LF conversion that we might not want.

Actually, if autocrlf=true is being used and git considers the file
text, then that is fine. It will be converted back to all LF before
writing to the repo.

But safecrlf=true would abort the commit, because the file would be
recreated with all CRLF (different from current file) on next
checkout. safecrlf=true ensures that nothing will be "corrupted."

> So the question is whether we consider that the possibility of
> people using text tools that can't handle CRLF properly in
> cygwin/MinGW outweighs the ability to move files around between
> environments.

No. This is something that every person using git has to consider
themselves (how they should set core.autocrlf) and nothing that we
do in our repo forces any user in any direction.

> I'll leave you guys decide on what you think should take precedence.

I am still happy to hear about any problems that anyone encounters
with my suggested solution, but the more ways that I describe how it
works, the more confident I get that it is the right thing to do.

//Peter

Peter Stuge wrote:
>> But safecrlf=true would abort the commit, because the file would be
>> recreated with all CRLF (different from current file) on next
>> checkout. safecrlf=true ensures that nothing will be "corrupted."

You're presumably already doing fancy redirection if you're using grep to
generate source code to commit, so no harm in expecting someone to add one
more pipe to the end, thru unix2dos.

Michael

On 2010.12.07 20:52, Peter Stuge wrote:
> I am still happy to hear about any problems that anyone encounters
> with my suggested solution, but the more ways that I describe how it
> works, the more confident I get that it is the right thing to do.

Provided that the git documentation about eol "setting" the text 
attribute is wrong (or need to be interpreted in a non obvious way) and 
the results I experienced in July when using "eol=lf" were just a fluke.

Regards,

/Pete

Ran a test with the following options (which, AFAIK are MSYS-Git's 
default) using my new megatest branch, which is based branched off 
libusb's very first commit.

For reference, since this is what I create as new .sh files, the output 
of my "git config -l" for the test can be found at [1]

I'm afraid I still managed to get CRLF when using both "eol=lf" and 
"text eol=lf" when I switched back.

o added .gitattributes ("eol=lf", no "text")
o added config.sh (LF) => config.sh is LF, autogen.sh is CRLF
o dos2unix autogen.sh and try to commit => nothing to commit (logical 
since already LF in repo)
o git checkout master
o git checkout megatest => both .sh files CRLF

o modify .gitattributes ("text eol=lf")
o added config2.sh (LF)
o dos2unix *.sh and try to commit => nothing to commit
o git checkout master
o git checkout megatest => all .sh files CRLF

I'll try with both autocrlf and safecrlf next, but I do understand that 
I was using MSYSGit's default settings for those here.

If you can identify what I did wrong here, or may have missed, let me know.

Regards,

/Pete

[1] 
http://git.libusb.org/?p=libusb-pbatard.git;a=commitdiff;h=ac15349eb9cd5e7196fd5fe4eec21d53cd88ff07;js=1

Well, using both safecrl=true and autocrlf=true is even worse.

changed safecrlf to true BEFORE creating/switching branch
added .gitattributes ("text eol=lf")
created config.sh (LF): "git config -l > config.sh"
git add config.sh
=> fatal: LF would be replaced by CRLF in config.sh

Now I'm starting to remember where that summer afternoon in July 
disappeared...

Might give a try to ____crlf input or off, but I think I'll wait for 
advice, in case you think there's something wrong with the way I'm 
running these tests.

Oh and for the record:
$ git --version
git version 1.7.1.msysgit.0

Regards,

/Pete

On 2010.12.07 23:53, Pete Batard wrote:
> o git checkout master
> o git checkout megatest => all .sh files CRLF
> (...)
> If you can identify what I did wrong here, or may have missed, let me know.

Thought that maybe switching to a master that has -crlf is the issue 
there, but I switch between 2 branches that have the same "*.sh text 
eol=lf", or a branch that that has no .gitattributes, it's still the same...

/Pete

Pete Batard wrote:
>> On 2010.12.07 23:53, Pete Batard wrote:
>> > o git checkout master
>> > o git checkout megatest => all .sh files CRLF
>> > (...)
>> > If you can identify what I did wrong here, or may have missed, let me
know.
>>
>> Thought that maybe switching to a master that has -crlf is the issue
>> there, but I switch between 2 branches that have the same "*.sh text
>> eol=lf", or a branch that that has no .gitattributes, it's still the
same...

Can you try switching to a branch that not only has no .gitattributes, but
also has no config*.sh?

Michael

On 2010.12.08 01:09, Michael Plante wrote:
> Can you try switching to a branch that not only has no .gitattributes, but
> also has no config*.sh?

The branch without .gitattributes was like this. It was the branch where 
I couldn't add config.sh, so I removed it.

/Pete

On 2010.12.07 19:23, Pete Batard wrote:
>> Why does that matter? GCC might handle CRLF line endings the way you
>> want; many other tools you can use with C files (grep, for one) don't.
>
> Mmmm, I hadn't really considered that, even though I seem to recall that
> Michael mentioned something like that. That's actually a good point. If
> we expect people to use tools that can't handle CRLF in cygwin and
> MinGW, then yes, I have to review my position.

Just tested grep, more (less), vim, and a few other command line tools 
in cygwin and MinGW, and at least for the versions I have, they are CRLF 
aware.
They will produce LF'd output however, so if we were to use grep to 
create a source, we'd get CRLF -> LF conversion that we might not want.

I would still expect all the default cygwin/MinGW text tools to be CRLF 
aware (they run on Windows, so they'd better be), but unless we find 
concrete evidence that any text tool from these environments (and for 
all the versions we are planning to support) handles any line 
terminator, and that everybody who creates a text utility for 
cygwin/MinGW is also supposed to handle both terminators properly (which 
is a big if), we're running the risk of breakage.

So the question is whether we consider that the possibility of people 
using text tools that can't handle CRLF properly in cygwin/MinGW 
outweighs the ability to move files around between environments.

As I hadn't considered the former, and the discussion has been going 
long enough, I'll leave you guys decide on what you think should take 
precedence.

Regards,

/Pete

Peter Stuge wrote:
>> Segher Boessenkool wrote:
>> > So we don't need a .gitattributes at all; at most we need to advise
>> > people to keep core.autocrlf off for our trees.
>>
>> The default is off, so I think most other projects simply assume that
>> people do not touch these settings.
>>
>> I suggested Pete and Michael to set them, so that they would get CRLF
>> line endings in their work trees, since they work on Windows.

The default for msysgit is autocrlf=true, safecrlf unspecified.  But it
gives you the option to select something else when you install; that's just
what happens if you blindly click "next>" all the way through.

Michael

Michael Plante wrote:
> >> > at most we need to advise
> >> > people to keep core.autocrlf off for our trees.
> >>
> >> The default is off,
..
> The default for msysgit is autocrlf=true, safecrlf unspecified.

Very significant! Thanks for noting this! When you mention it I
recall seeing that dialog during installation.

> But it gives you the option to select something else when you
> install; that's just what happens if you blindly click "next>" all
> the way through.

Which is likely that many do, certainly if they don't already know
how the options work, and in any case since there is a choice at all
and especially since autocrlf=true is the default for msysgit, then
it's fair to assume that some will have it set.

autocrlf=true means that worktree text files will have CRLF when
nothing else is said.

Thus autogen.sh won't run, and autoconf can't handle the multiline
AC_CONFIG_FILES. Adding .gitattributes with text eol=lf allows both
to work.

IMO this rules out the option to skip .gitattributes, since it would
mean that the default settings in msysgit would check out an unusable
autogen.sh (and configure.ac if multiline AC_CONFIG_FILES). Bad.

.gitattributes with text eol=lf checks out usable files. Good. This
is what we'll do.

Then it will be time for damage control, which can turn out to be
quite painless. I'm analyzing the issue that Michael has described,
and will post again when I'm done.

//Peter

On 2010.12.07 01:31, Peter Stuge wrote:
> .gitattributes with text eol=lf checks out usable files. Good. This
> is what we'll do.

Provided that it works always, while I *have* seen it fail to work.
And yes, I too hope that I somehow got it wrong in my testing, and it 
looks like the only way I will ever be able to convince you lot is if I 
can reproduce the issue again, and you can see it yourself, despite the 
fact that, in all logic, -crlf would solve our issue and wouldn't 
require anybody testing the wonderful mysteries of git's line ending 
conversions.

Oh, and I guess since there were a lot of crlf changes in git lately, 
even if I can reproduce my original issue, you guys will ask me to 
upgrade, because who knows, it might be a git bug, and then all we'll 
have to do is ask ALL our users to upgrade their git to a version that 
is less than 6 months old...

Again, going with -crlf would not require any of that (unless git 
introduced a bug then fixed it), but sure, once again, let's waste our 
efforts figuring out how git works when, with -crlf, we could move on to 
doing more productive things.

Regards,

/Pete

Sorry, snipped a couple other things I meant to respond to...

Segher Boessenkool wrote:
>> Yup, those files should be marked as binary -- we don't want to see diffs
>> of big auto-generated XML-like files anyway.

Well, the VC6 files are not XML. (*.dsp/dsw)

>> I still don't understand why in your configuration those are checked
>> out as CRLF, but other text files are just LF

Oh, they are all CRLF in Windows; I just don't use those files in Windows,
so no issue.  I do use them in Linux, but I have git configured differently
there.  I use gcc in Linux, and Microsoft in Windows, so I'm ok with those
files going whichever way you want (presumably LF, as I suggested).

Michael

>>> Yup, those files should be marked as binary -- we don't want to see
>>> diffs
>>> of big auto-generated XML-like files anyway.
>
> Well, the VC6 files are not XML. (*.dsp/dsw)

I said XML-like, not XML -- aren't those files a big semi-hierarchical
list of almost not manually edittable gobbledy-gook?

>>> I still don't understand why in your configuration those are checked
>>> out as CRLF, but other text files are just LF
>
> Oh, they are all CRLF in Windows; I just don't use those files in Windows,
> so no issue.  I do use them in Linux, but I have git configured
> differently
> there.  I use gcc in Linux, and Microsoft in Windows, so I'm ok with those
> files going whichever way you want (presumably LF, as I suggested).

So for you, the issue is that those files go weird for git, even though you
never touched them?  Ho hum.

Segher

Thread: Re: [Libusb-devel] CRLF in files checked out on Windows (Page 8)

A cross-platform library that gives apps easy access to USB devices

libusb-devel