RE: [Fault-injection-developer] How is the patched insmod making module_list available

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

yea, I've done a good job of confusing people. :->  Let me explain...

(NOTE: The explanation is really long, so stay with me.  At least this
       will give me a forum for expressing my thoughts while looking=20
       through the code base.)

When I start examining a project, I tend to take a bottom's up approach
where:
1. I immediately start greping through the source code to get a feel for =
the
design
2. I then start attempting to build the project
3. I then start hacking at the source to prove or disprove my understandi=
ng
4. After all of this I start reading design documents.

For people who tend to approach problems from a higher level looking down=
,
this=20
will seem very backwards.  All of the above is just an explanation for wh=
y I
am
doing what I am doing... you could say it's a method to my madness.

So... I'm looking through the FITH cvs tree, trying to get a feel for how
things
are laid out.  I observer that:

* There is a user space utility called ficl, that is rooted in
FITH/src/ficl/.
  I can see that it is communicating with some FITH kernel module via ict=
l()
on
  a character device.  For now that is all I care to know about ficl sinc=
e I

  believe our immediate work is in implementing the kernel side code in s=
uch
a=20
  way that the Linux kernel hacker community will accept.

* There is a kernel character device called fith_dm.o that is rooted in=20
  FITH/src/dm/.  After poking around some I can see:
  - this is a device that houses the kernel side logic used by other kern=
el=20
    side fault injection
  - the "dm" in the name means "Decision Maker".  I note to myself that
    maybe fi_logic or something on those lines would make a better name,
    but I feel that I should think about it some more.
  - all communication seems to be over ictl() calls

* There is another kernel character device called fith_fakeirq.o that is=20
  rooted in FITH/src/fakeirq.  After poking around some I can see:
  - The only thing this module does is export a function called
"fith_getirq"
    which is a replacement for "request_irq" function call.

* There are a bunch of other kernel modules in their own directories root=
ed
  under FITH/src/intcpt/.  After some searching I figure out that "intcpt=
"
means
  "interceptor", and that each of the directories under intcpt implement
some kind
  of kernel side code for intercepting the normal operation of kernel cod=
e
in order
  to insert a fault.
  - There is a module named "fith_aslt.o" rooted in FITH/src/intcpt/aslt/=
. =20
    + I manage to find a translation for "aslt" and the promptly loose it=
,
and make=20
      a mental note to find a better name.
    + The module seems to just export a function called "fith_ioremap" th=
at
is a=20
      replacement for "__ioremap".
  - There is a module named "fith_dbp.o" rooted in FITH/src/intcpt/dbp/.
    + I find that "dbp" means "Dynamic Binary Patch", and imediately star=
t
      looking for functionality the duplicates existing projects (like
dprobes.)
    + The only thing the module appears to do is register a bunch of exit
functions
      using the "hooks.c" implementation (otherwise known as "Kernel
Hooks".)=20
    + I make a mental note to myself that this is where FITH is dependant=
 on
      dprobes (or at least this is where FITH is dependent on deprobes
dependencies),=20
      and that this will be an immediate blocker to getting FITH to compi=
le
on=20
      2.5.x since I'm pretty sure the hooks.c implementation has changed=20
      dramatically (or just doesn't exist) in 2.5.x kernels.
  - There is a module named "fith_dr.o" that is rooted in
FITH/src/intcpt/dr/
    + It looks like this module does something like "fith_dbp", but it's
      not obvious what the purpose of this module is at first glance.
    + This module has the same hook.c dependencies as fith_dbp.o.
  - There is a module named "fith_ovr.o" that is rooted in
FITH/src/intcpt/override/
    + It appears to implement replacements for the normal input/output
functions
      like inb/outb and such.  The functions do not seem to be exported s=
o
I'm=20
      not sure how they are used.  I'm also confused by the fact that all=
 of
the
      standard input/output functions are either macros are inline
functions, so
      I'm not sure how a loadable kernel module could replace these
functions.
  - There is a module named "fith_pf.o" that is rooted in
FITH/src/intcpt/pt/
    + It looks like this module implements a kernel hook for page faults,
but
      I only casually glanced at the code.
    + I'm pretty sure dpobes does the same thing, and wonder if this isn'=
t
something
      that could be done using dprobes.

* There is a patched version of insmod rooted in FITH/fith_insmod/
   + After poking around I figure out that this is the entire
     modutils v2.4.6 package, and that the only file that has been=20
     touched (other then a bunch of CVS symbols that were changed
     because we imported the source to our own cvs tree) was
     FITH/fith_insmod/insmod/insmod.c
   + I get a good feel for what the hacked up version of insmod is doing,
and how
     it is using the kernel modules that I already discovered.

Ok, now I have some kind of a clue as to what this whole FITH source
tree is trying to do.  I compile the source on a RH7.3 system by followin=
g
the instructions in the README and don't run into any problems.  Now I st=
art
attempting to build on a RH8.0 system where I have been playing with a=20
2.5.44 kernel.

The first thing I trip on is the lack of a kernel hook implementation, so
I do a little research and find the whole kprobes vrs. dprobes discussion
happening on lkml.=20

I then start trying to build fith_fakeirq.o (FITH/src/fakeirq/) and run i=
nto
all kinds of problems with hard coded assumptions in the FITH build scrip=
ts.
Since I have already convinced myself that all of the FITH kernel could
should
build inside the Linux kernel tree, I move over fakeirq.c and the fith
header
files to the 2.5.44 kernel tree and do not give a second thought as to wh=
y
I was seeing all the strange compile problems when building within the FI=
TH
source tree.

The fakeirq.c code doesn't have the kernel hook dependencies as some of t=
he
other modules, so it seems like a good starting point.  I'm able to build
just fine, but have some symbol resolution problems.  A couple of these=20
are the symbols that the patch discussed in an earlier thread exports.  I
just hack the fith header file that depends on the both of these symbols
and now the only unresolved symbol is "module_list".

It takes me a while to figure out where module_list is coming from.  None=
 of
the FITH code implements module_list, so it's not a dependency on one of =
the

other FITH kernel modules.  Then I notice that a symbol named module_list=
 is
used in the insmod utility.  I suspect that maybe the hacked version of
insmod
provides the symbol so I try to load fakeirq.o with a patched insmod.

insmod fails because it can not open "/dev/fith", so I check out insmod.c
and verify that if insmod.c can open /dev/fith, it will continue with
loading
the kernel module.  insmod.c will attempt to do some dirty tricks on the
module
with the help of /dev/fith, but even if /dev/fith was really /dev/null
insmod
will still completely load the module and just print an error message abo=
ut
not
being able to insert a fault set.  So I trick insmod by making a /dev/fit=
h
node
with major and minor numbers for /dev/null, and I convince myself that I =
am
able=20
to load fakeirq.o without running into problems with unresolved symbols.=20

After looking into this some more this morning I realized that I was wron=
g,
and=20
that the patched insmod was not able to load the module, so I still do no=
t
know
now module_list was supposed to be resolved.  Maybe this was a symbol tha=
t
existed
in 2.4.x?

What a long, lingering email.... if you made it this far the I must
apologize for
this horribly crafted email. :-(

	-rusty

-----Original Message-----
From: Wang, Stanley=20
Sent: Thursday, October 24, 2002 12:56 AM
To: Lynch, Rusty
Cc: 'fau...@so...'
Subject: RE: [Fault-injection-developer] How is the patched insmod
making module_list available

I've been totally confused.
As I know, the fith_insmod depends on fith_dm, and fith_dm depends on all
interceptors (includeing fith_fakeirq).
So fith_insmod will return an error unless all FITH's modules have been
loaded into kernel.
Can you give me more information about the error you encounted, Rusty?

Thanks :)
Stanley
-----Original Message-----
From: Lynch, Rusty [mailto:rus...@in...]
Sent: 2002=E5=B9=B410=E6=9C=8824=E6=97=A5 10:52
To: 'fau...@so...'
Subject: RE: [Fault-injection-developer] How is the patched insmod making
module_list available

Actually, it's fakeirq.o, not fith_fakeirq.  (hmm... maybe the=20
build script that I tossed out the window is moving fakeirq.o=20
to fith_fakeirq.o? It doesn't matter... I'm just getting a feel
for the code.)

* NOTE: Keep in mind I hacked out the need for the symbols your
        kernel patch was depending on.

[rusty@penguin2 rusty]$ sudo /sbin/insmod fakeirq
Using
/lib/modules/2.5.44-ac2/kernel/drivers/faultinjection/fakeirq/fakeirq.o
/lib/modules/2.5.44-ac2/kernel/drivers/faultinjection/fakeirq/fakeirq.o:
unresolved symbol module_list

I can add all kinds of unresolved symbols to a module and fith_insmod=20
is happy to install the module.

	-rusty

-----Original Message-----
From: Zhuang, Louis=20
Sent: Wednesday, October 23, 2002 7:37 PM
To: Lynch, Rusty
Subject: RE: [Fault-injection-developer] How is the patched insmod
making module_list available

Rusty,
	Did you mean you can not insert fith_fakeirq by normal insmod?

    > -----Original Message-----
    > From: Lynch, Rusty=20
    > Sent: Thursday, October 24, 2002 10:17 AM
    > To: Zhuang, Louis; Lynch, Rusty
    > Subject: RE: [Fault-injection-developer] How is the=20
    > patched insmod making module_list available
    >=20
    >=20
    > What's there to log? /sbin/insmod fails because of an=20
    > undefined symbol, module_list.
    >=20
    > Although it looks like it has nothing to do with 2.5.44. =20
    > I think the patched insmod=20
    > just doesn't catch the fact that module_list is not=20
    > defined. Kind of a side effect?
    >=20
    > 	-rusty
    >=20
    > -----Original Message-----
    > From: Zhuang, Louis=20
    > Sent: Wednesday, October 23, 2002 7:11 PM
    > To: Lynch, Rusty
    > Subject: RE: [Fault-injection-developer] How is the patched insmod
    > making module_list available
    >=20
    >=20
    > We have not a try for 2.5.44... could you give us your error log?
    >=20
    >     > -----Original Message-----
    >     > From: Lynch, Rusty [mailto:rus...@in...]
    >     > Sent: Thursday, October 24, 2002 10:02 AM
    >     > To: 'fau...@so...'
    >     > Subject: [Fault-injection-developer] How is the patched=20
    >     > insmod making module_list available
    >     >=20
    >     >=20
    >     > I'm checking out the code and trying to get fakeirq.c to=20
    >     > compile with the
    >     > 2.5.44 kernel, but have an unresolved symbol for=20
    >     > "module_list" unless I use
    >     > the patched insmod.
    >     >=20
    >     > What's up with that?  It looks like all that was changed=20
    >     > in the insmod
    >     > source tree was insmod.c, and I don't see anything having=20
    >     > to do with
    >     > "module_list".
    >     >=20
    >     > 	-rusty=20
    >     >=20
    >     >=20
    >     > -------------------------------------------------------
    >     > This sf.net email is sponsored by: Influence the future=20
    >     > of Java(TM) technology. Join the Java Community=20
    >     > Process(SM) (JCP(SM)) program now.=20
    >     > http://ads.sourceforge.net/cgi-bin/redirect.pl?sunm0002en
    >     >=20
    >     > _______________________________________________
    >     > Fault-injection-developer mailing list
    >     > Fau...@li...
    >     > https://lists.sourceforge.net/lists/listinfo/fault-injecti
    > on-developer
    >=20

-------------------------------------------------------
This sf.net email is sponsored by: Influence the future=20
of Java(TM) technology. Join the Java Community=20
Process(SM) (JCP(SM)) program now.=20
http://ads.sourceforge.net/cgi-bin/redirect.pl?sunm0002en

_______________________________________________
Fault-injection-developer mailing list
Fau...@li...
https://lists.sourceforge.net/lists/listinfo/fault-injection-developer