subdirectory include config

  • Andre-Littoz

    Andre-Littoz - 2012-02-26

    Topic opened on behalf of  Zhang Qi

    REMINDER: never file a bug when you only need help. This is the place where you can get help and where anybody can contribute. Do not pollute the bug database. Thank you all.

    Original question:

    The first time I genxref the linux kernel, I found that if you want to locate some header file like asm/io.h, when you chose the arm arch, it should point to the arch/arm/include/asm/io.h. But I got nothing, and I found the path displayed on the page point to something like /linux/asm-arm/io.h which is a wrong path. So I review the lxr.conf about subdirectory section. I found that incprefix and maps may be wrong. Then I search the web and found a page . This seems can solve the problem. Now I got a quetion, that page says change the incprefix like , what the xxx means? Do it means all the arch that can be found or I need to specify a arch like x86 or arm? If I replace the xxx with $a ( the variable), will it work?

    I run the apache2 on centos, using the lxr-0.10.2

    The Linux kernel has a rather sophisticated directory structure to handle the different hardware it has been ported on. This directory structure has evolved with time and the historical comment in lxr.conf are no longer valid for the present kernel. The example in (Rewrite Rules for Include Paths) comes from a quick analysis of the present directory structure.

    Architecture specific include files are stored in /arch/port_name/include/
    Architecture independent inclue files are in /include
    This defines the content of incprefix which should be written:

    'incprefix' =>
              [ '/include'
              , '/arch/i386/include'
              , '/arch/ppc/include'
              , 'arch/arm/include'
    [i]... and all other hardware ...[/i]

    This is not convenient. You can use a short-hand notation in association with a rewrite rule.

    'incprefix' =>
              [ '/include'
              , '/arch/virtual/include'

    Include directives are coded in the kernel either as:

    #include <[i]subsystem[/i]/file.h>


    #include <asm/file.h>

    As you noticed, the /asm directory is deeply buried inside the /arch hierarchy. You must find a way to make


    point towards the right /arch/*/include directory.

    The previous


    will give:
    A- architecture independent include


    will hit in subsystem (with subsystem equal to linux, acpi, …).
    B- architecture dependent include


    but there is no


    directory. The word virtual must be transformed into something meaningful. If we suppose that variable


    has a


    attribute containing the list of all /arch subdirectories, the rewrite rule is:
    "_Replace word 'virtual' by each element in _


    list until we get a hit"
    But this rewrite rule must only be applied if we are testing an /arch subdirectory.  This is written as:

    'maps' => { '\/arch\/virtual\/' => '\/arch\/$a\/' }

    I use virtual here, while the article in uses xxx. Just replace the word in both places, its sole role is to create a level in the path hierarchy. However, if you change it, you must change it in two locations, which is error-prone. The rewrite rule may be modified as:

    'maps' => { '\/arch\/[^\/]+\/' => '\/arch\/$a\/' }

    which means "if you meet /arch followed by any name, apply the rule".

    While writing this answer, I thought of an even simpler incprefix/maps pair, but I did not test it to see if is is free of side effects:

    'incprefix' =>
              [ '/include'
              , '/arch/include'

    coupled with:

    'maps' => { '\/arch\/' => '\/arch\/$a\/' }

    Since virtual xxx are not used in the right side of the rewrite rule, there no point to add them in incprefix

    End of this rather long answer. Please give feedback if that solve your problem.

  • Zhang Qi

    Zhang Qi - 2012-02-27

    Thanks for your help.
    I genxref the linux kernel 3.2.7 twice. The first time I made some mistake so it failed. Today I re-genxref the source tree. This time I checked the config file twice and it succeed. I used your new incprefix-maps you mentioned but it didn't work.  But I changed the maps to 'maps' => { '/arch/' => '/arch/$a/' and it works quiet well. All the /arch/xxx/include/asm/yyy.h work pretty well. It can locate thoes header files. That's awesome. But I got a new problem.
    It can locate the .h files included using the absolute path like #include <linux/io.h> or #include <asm/xxx.h>, But, if it is using the relative path like #include " xxx.h" or #include " yyy.c" which xxx.h is in the same dir with the file inclcude it, it won't locate the xxx.h file.
    I have another question. I found that some identifier could not be linked automatically. But if you search them, you can find them in some .h files or .c files. And I found that, if aaa.c include bbb.h and bbb.h include ccc.h. An identifier or function declared in ccc.h could be linked automatically in the bbb.h or bbb.c but when aaa.c or aaa.h use the identifier or function, it  won't be linked automatically.

  • Andre-Littoz

    Andre-Littoz - 2012-02-27

    Problem 1: can't located included file in current directory
       Give me an example with name of current directory, name of including file, name of included file. This could be an adverse side effect of over simplified 'maps' rule.

    Problem 2: identifier not being recognised
      This is a known ctags issue. ctags does not capture all variables and functions. Its C parser is approximately good but not totally satisfactory. Its Perl parser captures only function names; you get identifier references (but not definitions) if they happen by chance in other languages.
       Give me name of some identifiers and language in which they are defined.
       Free-text search is a known work-around when language parser fails but is not an exact replacement.

    Problem 3: transitive inclusion
       I do not understand. Identifier definitions are captured in an independent pass of genxref. Then the references are collected in a following independent pass. It does not matter that a file is included or not; whenever a declared identifier is found, it is decorated with a <span class='fid'> tag. You can even get wrong decoration because LXR does not check the definition language. It does not either take consideration for any order of appearance.
       Give an example of good context and one of bad context.

  • Andre-Littoz

    Andre-Littoz - 2012-02-27

    Forgot to ask a question.

    How long does it take to index your 3.2.7 kernel? Give information about your computer: clock frequency, memory size, CPU type, OS name and version. Thanks.

  • Zhang Qi

    Zhang Qi - 2012-02-27

    For Q1,  please open /arch/arm/kernel/compat.c, see that

    0018 #include <linux/types.h>
    0019 #include <linux/kernel.h>
    0020 #include <linux/string.h>
    0021 #include <linux/init.h>
    0023 #include <asm/setup.h>
    0024 #include <asm/mach-types.h>
    0025 #include <asm/page.h>
    0027 #include <asm/mach/arch.h>
    0029 #include "compat.h"

    in my lxr page, the compat.h is not linked. And a new problem is the asm/mach/arch.h is also not linked.

    go to /arch/x86/kernel/asm-offsets.c
    in this file in my lxr page, the asm-offsets_32.c and asm-offsets_64.c are also not linked
    see line 0032~0035, like

     0033     OFFSET(TI_status, thread_info, status);

    I thought TI_status is a variable or macro define before, but I did a general search and found that it is a function.  This is not   linked either.  But

    0032     OFFSET(TI_flags, thread_info, flags)

    the TI_flags is also a function and it is linked.  This is weird. By the way, lxr is excellent,  maybe not powerfull enough, it is awesome.

    I install Centos in vmware8 under win7 ultimat.
    My pc has AMD X6 1090T 3.2G + 8G ram + 1T 7200rpm HDD. I give vmware 4 cores to run Centos. I didn't notice how long does it take to index the kernel. It seems take more than 3 hours.
    Thank you

  • Andre-Littoz

    Andre-Littoz - 2012-02-28

    Same for me if I enter /arch/arm/kernel/compat.c with default architecture setting, i.e. x86. Just set the 'a' variable (labeled architecture) to arm through a link or a menu item and everything becomes OK.

    If you want to read architecture-specific kernel parts, do not forget to set the 'a' variable (architecture) to the appropriate value.

    While trying to understand why it worked for me and not for you, I realised that correctly displaying and hyperlinking #include "generated-something" requires extra 'incprefix/'maps' with new variables. However I need to think a lot about it before suggesting rules because they seem to be rather complex.

    #include "compat.h"
    It looks like you discovered a bug. I investigate.

    /arch/x86/kernel/asm-offsets.c and #include "asm-offsets_32/64.c": same bug

    OFFSET and symbols not defined (consequently not hyperlinkable)
    Same for me. I did a "general search" with TI_status which did not find any explicit definition. TI_status is defined by OFFSET macro in a "concealed" way. With "concealed", I mean  definition tricky enough that ctags can't discover it. And if ctags does not report a definition, LXR won't know anything about that symbol. And the symbol is not hyperlinked.

    Remember that LXR is not a compiler. It looks only at "first level" textual appearance of symbols. It is good enough for the majority of projects. The Linux kernel case is an extreme one. It pushes gcc and its macro interpreter to its bleeding edge technical capacities. LXR cannot keep abreast with all the cute tricks in the kernel. This is why the "free-text search" feature is provided to work around its limitations.

    Nothing can be done, unless to use gcc itself as the internal LXR engine. The consequence would be catastrophic indexing and display performances.

  • Andre-Littoz

    Andre-Littoz - 2012-02-28

    Included files not hyperlinked from current directory

    This was caused by a too-naive rewrite rule in 'incprefix'/'maps' as I feared. The suggested

    'maps' => { '\/arch\/' => '/arch/$a/'

    rule blindly adds a nom path component. Correct if the '/arch/include' was added by an 'incprefix' item because the architectecture-specific part is missing. But when current directory is '/arch/x86/', the 'maps' rule gives '/arch/x86/x86/' (architecture part duplicated) which does not exist.

    Consequently, the correct 'incprefix' and 'maps' directives are:

      , 'incprefix' => [ '/include'
                      , '/arch/%VIRTUAL%/include'
      , 'maps' => { '\/arch\/%VIRTUAL%\/' => '/arch/$a/' }

    where I use %VIRTUAL% to mark the generated part. If %VIRTUAL% is a real directory, change that marker to anything not existing in the source-tree.

    There was no bug in LXR code. This illustrates how cautious you must be when writing 'maps' rules.

  • Zhang Qi

    Zhang Qi - 2012-02-28

    Thanks for your help!
    I changed the lxr.conf as your suggestion. it works. The header file or the c file can be located correctly and it works well.
    But this afternoon I found a new problem, yesterday I have found it but I didn't notice. See the file linux-3.2.7/​arch/​arm/​mach-at91/​at91rm9200.c

    0015 #include <asm/irq.h>
    0016 #include <asm/mach/arch.h>
    0017 #include <asm/mach/map.h>
    0018 #include <mach/at91rm9200.h>
    0019 #include <mach/at91_pmc.h>
    0020 #include <mach/at91_st.h>
    0021 #include <mach/cpu.h>

    those including asm/something works find but the mach/XXX.h is not hyperlinked.
    while, there is a include  directory in  the linux-3.2.7/​arch/​arm/​mach-at91/ directory. the mach is in the include dir.
    So this afternoon, I test several times, I realized that the incprefix/map can only find those header file in the path like

     /arch/xxx/include/        or        /root/include

    but, like the at91rm9200.c, it is under /arch/arm/mach-at91, it needs header file from the include dir in mach-at91.
    Your incprefix/map does not work on this condition. I tried change or add some regex expression in the lxr.conf, but it didn't work. So please help me again.

  • Andre-Littoz

    Andre-Littoz - 2012-02-28

    A quick answer: it is a simple matter of manipulation the include path with 'incprefix' and 'maps'.

    Step 1:
    Create a new variable like

      , 'arm_cpu' =>
         { 'name'  => 'Arm processor type'
         , 'range' => [qw(at91 bcmring clps711x cns3xxx davinci)]

    I put in 'range' only several partial directory names found in /arch/arm as _mach-/i]xxx/. Add the others if you want to be exhaustive. It is likely you need an equivalent trick for plat-xxx/.

    Step 2:
    Add a new include prefix

     , 'incprefix' =>
         [ '/include'
         , '/arch/%-VIRTUAL-%/include'
         , '/arch/arm/%-CPU-%/include'

    Step 3:
    Add a new rewrite rule to point to /arch/arm/mach-*/include/

     , 'maps' =>
         { '\/arch\/%-VIRTUAL-%\/' => '/arch/$a/'
         , '\/arch\/arm\/%-CPU-%\/' => '/arch/arm/mach-${arm_cpu}/'

    Run LXR, it works!

    I think an equivalent trick should be used for thr other architectures. A new feature should be added to LXR as "conditional variable", i.e. a variable valid only if another variable has a given value. I defined above variable arm_cpu to list the arm CPU variants. Obviously, I need the same for other CPUs. But it is much better to have an individual list per architecture because we could have duplicate entries and also many entries are not meaningful for a given architecture.

    I'll do that after 0.11 release._

  • Zhang Qi

    Zhang Qi - 2012-02-29

    Hello there
    I changed as you did. It indeed can locate the header files or c files. But I found a new bug.

            , 'armcpu' =>
                { 'name'  => 'ARM CPU type'
                , 'range' => [ readfile('/home/zhang/source/linux_kernel/mach') ]
            , 'armplat' =>
                { 'name'  => 'ARM Platform'
                , 'range' => [ readfile('/home/zhang/source/linux_kernel/plat') ]
        , 'incprefix' => [ '/include'
                , '/arch/%xxx%/include'
                , '/arch/arm/%ARM-CPU%/include'
                , '/arch/arm/%ARM-PLAT%/include' 
        , 'maps' =>
                { '\/arch\/%xxx%\/' => '/arch/$a/'
                , '\/arch\/arm\/%ARM-CPU%\/' => '/arch/arm/mach-${armcpu}/'
                , '\/arch\/arm\/%ARM-PLAT%\/' => '/arch/arm/plat-${armplat}/'

    the file mach and plat contains arm cpu and platform dir name.  this is ok , it works.
    The bug is when I open /lxr/source, it look good, but when I chose ARM arch as arm or some else, and hit change, the bug comes.  On the top, the same row with Architecture and Version select column, also there are ARM plat and ARM arch, there is a new select column which name has nothing but colon and it a blank select column.
    For example, if I select arm , s5pv210, samsung, and click change, I am sure lxr know what I choosed because I can see the result from the url like


    but the page display that nothing changed except arm arch. the arm cpu still is at91 and arm plat also iop, like I didn't click the change button. I test these a lot of time and I still don't know what wrong.
    I checked  the html source file after change something and click change button, I found something strange like

                        <td class="banner"><span class="banner"><a class='banner' href="/lxr/source/?a=arm; Linux kernel release 3.x <>
    These are the release notes for Linux version 3.  Read them carefully,
    as they tell you what this is all about, explain how to install the
    kernel, and what to do if something goes wrong. 
    WHAT IS LINUX?=samsung">linux-3.2.7</a>/</span></td>

    and before I click the change button, this part of cold should like

        <td class="banner"><span class="banner"><a class='banner' href="/lxr/source/?a=arm">linux-3.2.7</a>/</span></td>

    below is an url, that is copied after I click the change and open a hyperlink


    the Revert button also does not work.

  • Andre-Littoz

    Andre-Littoz - 2012-02-29

    Revert button:
    Its "revert" capability is very limited. As the doc says: "it reverts the variables to the value they had on entering this view." You "enter a view" either by clicking on a link or by clicking on the "Change" button. Once you click on this button, changes in variables take effect and "revert" will give you these new values.

    Revert is only intended to cancel changes before clicking on "Change".

    Faulty URL:
    Something is wrong because text has somehow crept into the 'href'. I can't diagnose without further information. Send me at ajlittoz (at) users (dot) sf (dot) net:
    - copy of your lxr.conf,
    - name of file you displayed when you got such a strange link in 'banner'
    - if possible, screen snapshot (mainly header area of LXR window) so that I can see the 'variables' row.

  • Zhang Qi

    Zhang Qi - 2012-03-01

    Hi, could you give me another email address pls? I have sent you an email to the address you give me, but it failed. I don't why, a reply msg says DNS could not be found.

  • Andre-Littoz

    Andre-Littoz - 2012-03-02

    Try page74010-sf (at) yahoo (dot) fr
    Replace the (…) by @ and . respectively. This is a robot protection measure. If you did not replace the characters in ajlittoz (at) users (dot) sf (dot) net, the address was not valid.

    Sorry for the delay, my Internet connection has been down for one day and a half.


Log in to post a comment.