Re: [Dmtcp-forum] Fault injection for DMTCP file descriptors

[Kapil A. <ka...@cc...>]

Hi Richard,

I almost forgot that we had _real_read and _real_write definitions :-).
Anyways, the wrapper code looks correct, although you shouldn't need to do
the saved_errno stuff. It is taken care of inside the DMTCP_XXX macros.

I am a little confused about the dmtcp_is_running_state() always returning
1. Is it possible for you to send me a stacktrace of the write call that
originates from DMTCP? That might help us narrow down why is it returning
1.

Thanks,
Kapil


On Tue, Jul 2, 2013 at 5:50 AM, Richard Potter <pot...@gm...>wrote:

> Hi Kapil,
>
> Thank you for your reply and the new info.  I tried a quick test of
> dmtcp_is_running_state() (using #include "dmtcpplugin.h") and the
> result was that it always returned 1, even when write() was sending
> out real application data, so it does not seem to help us.
>
> In regards to _real_write and _real_read, connecting gdb to a process
> shows this:
>
> Loaded symbols for
> /home/knoppix/dmtcp-1.2.7/dmtcp/src/../../lib/libmtcp.so.1
>    0x401a803c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
>    (gdb) br _real_write
>    Breakpoint 1 at 0x400bd690: file syscallsreal.c, line 459.
>    (gdb) br _real_read
>    Breakpoint 2 at 0x400bd600: file syscallsreal.c, line 454.
>    (gdb)
>
> And syscallsreal.c:439 contains:
>    LIB_PRIVATE
>    ssize_t _real_read(int fd, void *buf, size_t count) {
>      REAL_FUNC_PASSTHROUGH ( read ) ( fd,buf,count );
>    }
>
>    LIB_PRIVATE
>    ssize_t _real_write(int fd, const void *buf, size_t count) {
>      REAL_FUNC_PASSTHROUGH_TYPED ( ssize_t,write ) ( fd,buf,count );
>    }
>
> For reference, our attempt at wrapping write() basically looks
> like this:
>
> ssize_t write(int fd, const void *buf, size_t count) {
>   ssize_t rr;
>   if ( our filter code to identify which writes are really
>                        done by the application  ) {
>      our code to change the behavior of write
>      }
>      WRAPPER_EXECUTION_DISABLE_CKPT(); // The lock is released inside the
> macro.
>      rr = _real_write(fd, buf, count);
>      int saved_errno;
>      saved_errno = errno;
>      WRAPPER_EXECUTION_ENABLE_CKPT();
>      errno =saved_errno;
>      return rr;
> }
>
> The good news is it almost works.  The bad news is we are really just
> guessing here what is possible, based on a quick study of other wrappers
> in socketwrappers.cpp.  In fact, when you first said that _real_write
> were not defined, it sounded plausible, because we had not checked.
> It just compiled and linked by magic. :-)
>
> The dmtcp_is_running_state() would have been nice because our
> filtering code is matching for patterns, which works, but could break
> anytime. Therefore, any further clues on how to use
> dmtcp_is_running_state()
> or something else would be helpful.  Also, how lucky are we that the
> wrapping of write() is working?  Hints on how to do it more properly would
> be appreciated.
>
> DMTCP is amazing.  Wish I had time to dive in and really understand
> how it works.
>
> --Richard
>
> (It turned out that a problem we thought was related to the the write()
> wrapper
> occurs on unmodified dmtcp, so I'll send that in a separate email.)
>
> On Tue, Jul 2, 2013 at 1:43 AM, Kapil Arya <ka...@cc...> wrote:
> > Hi,
> >
> > DMTCP does not put wrappers around read and write and thus no _real_
> > versions of these functions. In your application, are the _real function
> > defined in the binary or in some shared library?
> >
> > A simple trick would be to use the function:
> >   int  dmtcp_is_running_state();
> > to check if the computation is in _RUNNING_ state or not. DMTCP uses
> > read/write only when it's _not_ in RUNNING state, i.e. during checkpoint
> and
> > restart.
> >
> > Does this help?
> >
> > Kapil
> >
> >
> >
> > On Thu, Jun 27, 2013 at 3:44 AM, Cyrille Artho <c....@ai...>
> wrote:
> >>
> >> Hi all,
> >> We are trying to use DMTCP for fault injection on sockets that are
> >> accessed via a file descriptor.
> >>
> >> The problem we ran into is that DMTCP sends its own data over existing
> >> application sockets. That data is used for internal bookkeeping in
> >> DMTCP, and not seen by the application.
> >>
> >> However, our wrapper for read sees this extra data, and we are thinking
> >> about how to ignore it or filter it out.
> >>
> >> Is it intentional that DMTCP uses "read" instead of "_real_read" for its
> >> own communication? If so, is there any chance for us to have internal
> >> use of "read" flagged somehow? It seems DMTCP now uses a lock for
> >> internal operations, so if DMTCP's internal use of "read" cannot be
> >> changed to "_real_read", the following may do the trick for us as well:
> >>
> >> lock
> >> set flag
> >> read
> >> clear flag
> >> unlock
> >>
> >> This way, we always know when the application uses read (the flag is not
> >> set).
> >>
> >> If there is no easy way for us to tell if DMTCP or the application calls
> >> "read", then we can try to filter the messages from DMTCP. They seem to
> >> use particular messages and port numbers. Is that documented somewhere?
> >> Is that expected to be stable?
> >> --
> >> Regards,
> >> Cyrille Artho - http://artho.com/
> >> Those who will not reason, are bigots, those who cannot,
> >> are fools, and those who dare not, are slaves.
> >>                 -- George Gordon Noel Byron
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> This SF.net email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev
> >> _______________________________________________
> >> Dmtcp-forum mailing list
> >> Dmt...@li...
> >> https://lists.sourceforge.net/lists/listinfo/dmtcp-forum
> >
> >
>