On Mon, Jul 14, 2008 at 1:05 PM, Subrata Modak <subrata@linux.vnet.ibm.com> wrote:
Dear Hisashi San,

Since we have already embarked on our path to mutual co-operation in
Linux testing and test cases development, i would like to discuss some
of my ideas, which i think is capable of adding shot-in-the arm for
linux testing.

I want to talk about Test Driven Development for Linux Kernel Testing,
which can bring us very early effective results and can catch Bug(s), if
any, in the kernel code much much early, and even before the code makes
it´s way through the stable kernel release.

The idea is to submit the test cases to LTP, for certain kernel
features, which makes its way through the kernel-mm tree. When a
developer submits his/her feature to mm-kernel, he gives a specification
of the features. Depending on the feature specification available either
from the developer himself, or, from manpages, if it is available that
soon, tests cases will be written.
From now on, I'll be agitating more to get man pages provided more with new syscalls and ther kernel-userland interfaces.  That will mean either I twist developers arms to write pages ;-), or I write them myself, with help from them.  I do think that man-pages, if well written, are often sufficient as (or at least a very good base for) a test specification.  Here's an example that I did with the timerfd API, finding two bugs in the process: http://thread.gmane.org/gmane.linux.kernel/613442 .  I did something similar while writing the utimensat(2) man page, finding 5 or 6 different bugs in the end, see http://linux-man-pages.blogspot.com/2008/06/whats-wrong-with-kernel-userland_30.html .
Now, we have an inherent question,
who writes and submits those test cases ?? Either it can be the
developer himself, or, somebody from the test community.
Or, IMO, preferably both, working together.
Since, the
developer is not bound to submit any test case,
I can't help but feel that part of the solution here is to agitate more so that the developer is (strongly) encouraged to submit test cases along with patches that change the kernel-userspace interface.  As things stand, there are far too many bugs in released interrfaces.
we can assume we need to
initiate doing something in this regard.

Having said that, i want to propose that, since in your project you have
already taken up the Job of making available the system call test cases
up-to-date, we can score a point here. Whenever a new system call code
enters the mm-tree in the kernel, somebody in your project can write the
test case(s) immediately and make it available to:

1) Your Project,
2) LTP,
3) LKML,
I would also be very happy to see any test cases you produce for new system calls.  What list do I need to subscribe to?
We will use the test case(s) to test that particular syscall feature and
report Bug(s) to the developer(s) and LKML. Meanwhile, updates to the
test cases will also go on depending on feedback from community. With
that we will address the following issues:
1) Catch bug at the earliest for the new syscalls in the kernel, so that
the cost of Bug fixing, when something is discovered later, can be
hugely reduced,
Bug fixing is perhaps the lesser problem.  The greater problem is that if there are bugs, we must change the ABI to fix them, and if that is done after a stable kernel release, then userland programmer's can end up having to do things like special case their code according to the kernel version.
2) We have a regular and systematic flow of test cases for both LTP and
Crackerjack. This process becomes institutional and gurantees us maximum
and continued code coverage for syscalls in future,
3) LTP-mm tree is born,
4) Others/developers get motivated to submit test cases(to LTP and
Crackerjack) at the time of submission of kernel features to
kernel-mm-tree, if we show them the benefit of early testing and it´s
effectiveness of controlling more bugs to seep in before it is too late
to fix.

For this to happen, we need to make our test cases ready when the
corresponding kernel features are in -mm tree. Let me know your ideas on
Part of the problem here is knowing when interface changes have occurred.  Seeing new system calls in a release is easy. But, for the project to be truly effective, all kernel-userland interfacechanges need to be checked for, including
system calls
/sys (okay -- that's a big ask at the moment)
virtual /dev interfaces used for kernel-uerland communication
And probably other things I forgot right now.
On Mon, 2008-07-14 at 13:39 +0900, hisashi.hashimoto.wh@hitachi.com
> Subrata,
>  >Masatake is already working to port the missing syscall test cases in
> >LTP from the Crackerjack Project to LTP. And it is already a huge
>  I am glad to hear that.
> >effort. I think the idea here is great as we are not re-inventing the
> Yes, you can do this and we can use our time to develope more test cases.
> >to port from you, unless somebody in LTP community has already submitted
> In that case, we will refer your cases and make them better.
> >https://lists.sourceforge.net/mailman/listinfo/ltp-list
> I joined now. I am very surprised that your community are
> working so active. I will check and I will post anything
> I can collaborate.
> >We will be collaborating as far as we can, as all of us striving to
> >Linux better.
> Yes, agree.  We will continue this effort for our purpose, Linux better.
> Please post anything to both our mailing list and cc-d your mailing list
> when you have to contact me.
> Thanks,
> Hisashi
> >送信者 : Subrata Modak <subrata@linux.vnet.ibm.com>
> >主題 : Re: Re[2]: Re[2]: Re[2]: Re[2]: [Crackerjack-devel] [LTP]Crackerjack andLinuxTes
> >受信日 :08/07/12 04:10
> >属性 : なし
> >
> >Dear Hisashi Hashimoto San,
> >
> >Thank you for writing back to us. We are delighted to colloborate on all
> >fronts to make Linux testing better. My comments embedded in the
> >following lines.
> >
> >On Fri, 2008-07-11 at 13:53 +0900, hisashi.hashimoto.wh@hitachi.com
> >wrote:
> >> Subrata,
> >>
> >>   Sorry for long time no reply.
> >> We had internal meeting of our project
> >> regarding how to proceed.
> >> Current our plan is:
> >> (1) Unfortunattelly, our test cases do not cover all system call yet.
> >>      Total number of system call in Kernel 2.6.26 is almost 326.
> >>       But, we developed around 299.
> >
> >Masatake is already working to port the missing syscall test cases in
> >LTP from the Crackerjack Project to LTP. And it is already a huge
> >effort. I think the idea here is great as we are not re-inventing the
> >same wheel. We can use your test cases released under GPLv2 and port to
> >LTP format.
> >
> >We gain by getting more kernel test cases in LTP without the need to
> >massively write things. And you retain your goal of providing a
> >regression comparisn framework. In future, whenever the no. of syscalls
> >goes increasing in kernels, and as soon as you write them ,we will try
> >to port from you, unless somebody in LTP community has already submitted
> >a test case in that regard. Or, even we can concentrate to write test
> >cases in some other are as well.
> >
> >Presently, the first job is to port the existing ones from you to us.
> >
> >>      Therefore,  we would like to start developing in soon.
> >>      The starting date would be some time in August.
> >> (2) Some of our developed test cases do not meet our quality criteria.
> >>      As you know, we set our quality goal as covarage ratio more than
> >>      yours.
> >>        We are now checking all tests again. And if necessary, we need to
> >>      enhance test cases.
> >
> >Good to know that. As and when you do this, we might also relook into
> >ours and change accordingly.
> >
> >> (3)  Through our acitivity (1) and (2), we would like to exchange
> >>       information and technical/engineering  discussion on this mailing list.
> >
> >Definitely yes. All of you can subscribe to the ltp mailing list:
> >https://lists.sourceforge.net/mailman/listinfo/ltp-list
> >
> >And please add us as CC to all the technical discussion you have. Guys
> >in ltp mailing list will need to subscribe yours if they want to reply
> >back to you.
> >
> >>
> >> Yesterday, 4 Maintainers(Andrew Morton, Paul Moor, James Morris, Thomas Gleixner)
> >> visit Japan to make presentation regarding the latest kernel/security activity.
> >> I had opportunity to make presentation to them about our current status.
> >> I present that
> >> (1) We released test cases 300/326.
> >> (2) We plan to make complete set and enhance quality.
> >> (3) We continue to talk with other test project(LTP as you, AutoTest:Martin Bleigh),
> >>      and exchange information and status, and others.
> >> Maintainers said/recommend to expand the target, to continue the effort
> >> and collaborate with LTP and so on.
> >
> >We will be collaborating as far as we can, as all of us striving to
> >Linux better.
> >
> >
> >Regards--
> >Subrata
> >
> >>
> >>  I hope you and we can do the collaboration work on test project.
> >>
> >> Best,
> >>
> >> Hashimoto Hisashi
> >
> >
> >
> Hisashi Hashimoto
> Section Manager
> Open Source Software Promotion Center
> Platform Software
> Hitachi, Ltd., Software Division
> Tel : +81-45-862-8424, Fax : +81-45-862-9047
> hisashi.hashimoto.wh@hitachi.com

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html