#187 wrong line numbers for tags in files with random ^Ms

[Ray]

Hi, Ctags Developers,

I am so proud to be an Exuberant ctags user, and I have been using it for about 2 years. It is working so far so good. However, I found if the file includes some "^M" in it, in my case the header files, which is "0x0d" in windows/msdos system. The Ctags will tread it as a line break, and the following "#define SOME" will totally be affected.

For example
/*---------------------------------------*/
1 #include <stdio>
2
3 void main()^M
4 {
5 printf("Hello World!\n");
6 }
7
8 #define SOME
9 #define OOK
/*---------------------------------------*/

When we build the "tags" file by "ctags -R .", the tag
for SOME and OOK will be all WRONG!, because there is "^M" in line #3.

Please note, ^M in VIM is not just two characters, "^"+"M", it is <CTRL-V>+<CTRL-M>, in hex it is 0x0d.

When we deal with a large project with a lot of source codes, if someone put the ^M in their files from different systems, Ctags will get the wrong tag positions. To make this easier is just to make some condition in Ctags, if this is the MS-Windows system, then don't treat "^M" as a line break, and count it as a line number, even though it should be counted in MAC OS.

Thanks for your attention!
Ray,

[Elliott Hughes]

Logged In: YES
user_id=1127237
Originator: NO

works for me with the current version of Ctags in subversion:

lithium:~/Projects/ctags/trunk$ cat -nv Test/bug1773926.cpp
1 #include <cstdio>
2
3 void main()^M
4 {
5 printf("Hello World!\n");
6 }
7
8 #define SOME
9 #define OOK
lithium:~/Projects/ctags/trunk$ ./dctags -n -f - Test/bug1773926.cpp
OOK Test/bug1773926.cpp 9;" d file:
SOME Test/bug1773926.cpp 8;" d file:
main Test/bug1773926.cpp 3;" f
lithium:~/Projects/ctags/trunk$

works with ctags 5.6, too:

lithium:~/Projects/ctags/tags/Ctags-5_6$ ./ctags -n -f - ../../trunk/Test/bug1773926.cpp
OOK ../../trunk/Test/bug1773926.cpp 9;" d file:
SOME ../../trunk/Test/bug1773926.cpp 8;" d file:
main ../../trunk/Test/bug1773926.cpp 3;" f
lithium:~/Projects/ctags/tags/Ctags-5_6$

the relevant code doesn't seem to have changed since Ctags 5.1, which is the oldest i have available.

[Ray]

Logged In: YES
user_id=1867368
Originator: YES

As I emphasized, if you just simply copied the code I provided, then
everything works fine, but, when you use Vim editor, in windows system
to insert some special characters, like
, then the problem will
appear,

I have tried this many times, and it is absolutely reproduceable.

[Elliott Hughes]

Logged In: YES
user_id=1127237
Originator: NO

the output i pasted below was "cat -v", not "cat". that file *does* have a control-M in it.

[Ray]

test file for your testing,

[Ray]

Logged In: YES
user_id=1867368
Originator: YES

I uploaded a file for your testing, my environment is windows xp, please use your ctags.exe to see if this error can be produced or not, thanks!
File Added: test.c

[Elliott Hughes]

Logged In: YES
user_id=1127237
Originator: NO

good idea. can you show me the command-line you're using, and attach the resulting tags file too, please? also, the output of "ctags --version" so i can see exactly what you're running.

[Ray]

Logged In: YES
user_id=1867368
Originator: YES

Per your request,

Command Line:
-------------
ctags.exe -R .

Version Info:
--------------------
Exuberant Ctags 5.6, Copyright (C) 1996-2004 Darren Hiebert
Compiled: Aug 13 2007, 22:41:24
Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
Optional compiled features: +win32, +internal-sort

[Ray]

Logged In: YES
user_id=1867368
Originator: YES

And I also tried the original build

Exuberant Ctags 5.6, Copyright (C) 1996-2004 Darren Hiebert
Compiled: Jul 30 2006, 16:12:20
Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
Optional compiled features: +win32, +regex, +internal-sort

[Elliott Hughes]

Logged In: YES
user_id=1127237
Originator: NO

here's the content of the submitted file:

lithium:~/Projects/ctags/trunk$ cat -vn ~/Desktop/test.c
1 #include <stdio.h>^M
2 ^M
3 #define ERROR_HAPPENED^M 50 ^M^M^M
4 #define OK 2^M
5 #define NEXT_DEFINE 3^M
6 ^M
7 int main(int argc, char* argv[])^M
8 {^M
9 printf("Hello world\n");^M
10 return 0;^M
11 }^M
12 ^M
lithium:~/Projects/ctags/trunk$

and here are the generated tags:

lithium:~/Projects/ctags/trunk$ cat tags
!_TAG_FILE_FORMAT 2 /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED 1 /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_PROGRAM_AUTHOR Darren Hiebert /dhiebert@users.sourceforge.net/
!_TAG_PROGRAM_NAME Exuberant Ctags //
!_TAG_PROGRAM_URL http://ctags.sourceforge.net /official site/
!_TAG_PROGRAM_VERSION 5.6b1 //
ERROR_HAPPENED /home/elliotth/Desktop/test.c 3;" d file:
NEXT_DEFINE /home/elliotth/Desktop/test.c 8;" d file:
OK /home/elliotth/Desktop/test.c 7;" d file:
main /home/elliotth/Desktop/test.c 10;" f
lithium:~/Projects/ctags/trunk$

the problem is caused by line 3, which contains a ^M in the middle of the line, and multiple ^Ms at the end of the line.

i'm not sure we can fix this cleanly. i've never seen a Mac OS 9 file (just '\r's) in real life, but then i'd not seen a file as random as the attached "test.c" before, either. i'm going to reassign this to Darren because he'll have to decide.

[Ray]

Logged In: YES
user_id=1867368
Originator: YES

Thanks to your effective responses, and hopefully this problem is going to be completely fixed, because I found it in my projects files, of course written by someone else, and I got a little doubt about ctags until I found this was caused by ^Ms.

[Darren Hiebert]

Logged In: YES
user_id=38016
Originator: NO

I changed the newline canonicalization to process a lone CR as a newline only when running on MacOS 9.
File Added: bug1773926.diff

[Darren Hiebert]

Patch to correct bug