Thread: [linuxsh-dev] [RFC] LinuxSH code reorg

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Introduction
============

I've been working with the LinuxSH code for about 10 months now, and in
that time I've come up with a few gripes and ideas concerning the core code
base and the methods developers use to structure and write drivers.  This RFC
attempts to address some of the challenges I've faced writing code for 
LinuxSH, and it hopefully provides some constructive ideas that we can all
use to better the code.  To a lesser extent, it asks about the future of 
LinuxSH, in regards to the imminent release of 2.5.x.

This RFC doesn't cover the specific APIs that would allow for a total code
rewrite across all SH boards/platforms (I'm still working on those :), but
it does talk about driver placement, arch/sh file organization, code reuse,
and developers not being lazy when writing their code.

Some of these ideas are taken from the Linux/MIPS port.  The MIPS port has
thus far been a good model for writing embedded systems code under Linux,
most embedded platforms have homes under this tree.  With the formation of
the linux-mips tree (http://sf.net/projects/linux-mips), their focus is now
on getting additions and patches tested and in the tree in a timely
fashion.  I believe it's beneficial to share concepts and code with MIPS
rather than architectures that aren't traditionally suited for embedded
boards, such as the i386.

Current Situation
=================

Inconsistency
-------------
Looking over the arch/sh subdirectory, I find a number of things disturbing
about the lack of consistency when targetting new boards.  There are two
toplevel board directories, overdrive and stboards.  overdrive appears to
contain code for the support of a single board, as does stboards.  The
remaining boards supported by LinuxSH are in various files in the kernel
directory.  Now, the new-machine.txt document in Documentation/sh does
provide a starting point for adding a new SH board, but it doesn't talk
about where to place other support code for the board, and it fails to
mention the setup_<board>.c convention that the majority of boards follow.

The result of placing board definition and support code in the kernel
directory is that the directory appears cluttered, and this will only
worsen as more boards are added.  This also makes maintenence combersome,
and discourages new developers from hacking on code (I say this because
it's discouraged me a bit :).

Another point of inconsistency was in the addition of the Maple Dreamcast
drivers.  Originally, they were placed in drivers/dreamcast, which
completely goes against standard kernel semantics of driver placement.  The
Maple Bus is an input bus within the Dreamcast, with a variety of
character-style input drivers, such as a mouse, keyboard, or joystick.
Normally, system buses go into a drivers/<bus> subdirectory, and bus
devices go under the respective driver interfaces they support, e.g. the
Maple joystick device driver would live under drivers/char/joystick.

All of the other device drivers that I've seen contributed by LinuxSH
developers have followed the normal convention, so that was probably an
isolated incident.  The misplaced Maple drivers have since been fixed.

Code Cleanliness
----------------
I don't want to take the time to site specific examples, because it would
take a while :).  But overall (just me manually viewing each file) the
source files in arch/sh leave a lot to be desired.  It looks like a lot of
source come from developers who come from embedded backrounds, not
bothering (or not knowing) about the kernel development guidelines.  Of
course, I could be wrong, and it's just laziness.  I'm guilty of it too.

Proposal for Change
===================

Review
------
All code that is submitted to go into LinuxSH should be reviewed by all
developers, where it's assumed that all interested LinuxSH developers are
subscribed to the linuxsh-dev mailing list.  If, after a 1-2 day period no
one objects to the inclusion of the patch, and the submitting developer has
write priveledges, she should feel free to apply the patch.  If someone
does object to the patch, however, then those two developers need to find a
solution, and if that still fails, the group should arbitrate or NIIBE
should step in and make the final call.  The important thing is that *all*
code, no matter how trivial, should be reviewed by the entire community.

Drop-in Tree
------------
While talking to a linux-mips developer while updating the latest revision
of LinuxSH, I came to a realization.  NIIBE spends a significant time
merging changes from upstream back into LinuxSH.  Also, hosting the entire
kernel and updating it is a substantial drain of space and bandwidth,
especially since LinuxSH affects such a small portion of the kernel.
Besides a few device drivers underneath the drivers/ tree, all LinuxSH code
lives under arch/sh and include/asm-sh.

The linux-mips project and ruby project (just two projects I've encountered
that do this) use "drop-in trees".  A drop-in tree only contains the code
relevant to the project, so for LinuxSH, its drop-in tree would look like:

  linux/Documentation/sh/...
  linux/arch/sh/...
  linux/include/asm-sh/...
  linux/drivers/char/sh-sci.[hc]
  linux/drivers/char/ec3104_keyb.c
  linux/drivers/char/maple*
  linux/drivers/char/joystick/maplecontrol.c
  linux/drivers/maple/...

So a developer would download the latest kernel and relevant patches, then
download the LinuxSH drop-in tree.  She runs a script called treelink.sh
that symlinks the files from the drop-in tree on top of the upstream
kernel.  She can then build the kernel for her platform.

This should also make NIIBE's job a lot easier, since he no longer would
have to do branch merges with upstream, but only worry about the overall
"sync" of the LinuxSH tree with the upstream tree.  If there isn't a
kernel-wide change that breaks LinuxSH functionality, then the tree remains
synced.

Drop-in trees also have the benefit of being small (a lot easier to
shuffle than a 136M tree, IMO), and helps developers to be more focused.

No Divergence
-------------
Haven given it minimal thought, it makes no sense for the LinuxDC project
to maintain another divergent tree, especially since the DC is just another
embedded SH board, and SH development is centralized around LinuxSH.  So
all DC drivers will be added directly to the LinuxSH tree.  LinuxDC would
still remain to provide user support, but there's now a central place to
grab and maintain the source.  All SH boards and their drivers should live
within LinuxSH.

Another point about divergence: I've seen/made some changes to the
drivers/net/8139too.c driver so that it can support the Dreamcast's
Broadband Adapter.  If we need to make changes to code that we don't
maintain, we should make the effort to contact the maintainer to ensure that
the code makes it upstream.  If we don't do this, upstream will end up with
shoddy support for our boards, i.e. as it stands the 2.4.7 8139too driver
won't work properly with the DC BBA.

Consistency
-----------
We need to come up with the Standard way of adding new boards to the tree,
while cleaning up the existing mess.  The format that linux-mips finalized
on (and I think it fits just as well with LinuxSH) is this:

- Only processor, kernel, and processor module code should go into
  arch/sh/kernel.  This directory should be "holy ground" - no
  board-specific hacks belong here.

- System controllers, like the hd64465 and hd64461 (please help me out with
  others) that share common code between multiple boards should live in an
  arch/sh/<system controller> directory.  Only boards that share a common
  system controller (does the 7750S fall into this category?  Or is that
  processor-specific code?) should have a system controller placed there.

- Each individual board that LinuxSH runs on should live in
  arch/sh/boards/<board>.  The actual "reference" name of the board should
  be used, not the marketing name.

So, a quick example of the new layout (a lot of files/boards ommitted):

  arch/sh/kernel/
  arch/sh/hd64465/
  arch/sh/hd64461/
  arch/sh/solutionengine/
  arch/sh/boards/hp620/
  arch/sh/boards/hp680/
  arch/sh/boards/hp690/
  arch/sh/boards/dreamcast/
  arch/sh/boards/cqreek/
  ...
  arch/sh/mm/
  arch/sh/lib/
  arch/sh/boot/

I also think we need to split off PCI support into a seperate subdirectory,
i.e. pci/, but I'm still working on that proposal so that can wait.

Cleanliness
-----------
C'mon folks, it's not that difficult to follow the guidelines set forth in
CodingStyle and other documents that describe kernel development semantics.
I have also been guilty of not doing this, and we need to make a concerted
effort to make sure our code is clean.  I know that most of us are embedded
systems developers, who may not be as familiar with kernel development, but
it's important that we strive to write high-quality code, especially since
this all goes upstream, and once that happens, your support base gets much,
much larger.

Garbage In, Garbage Out - Just because a piece of code is "functional"
doesn't mean it belongs in the tree, especially if it suffers from critical
design flaws or doesn't follow kernel guidelines.  We should be able to
police ourselves, and we should never settle for shoddy code.

A good place to look for kernel guidelines is, uh, in the kernel
Documentation directory itself.  CodingStyle, kbuild/, DocBook/, and the
files of the subsystems of any drivers we develop should be consulted
before we do *any* code or design.

I've always looked at kernel sub-ports like LinuxSH as "role models" for
young developers like myself who are looking for a fit in a smaller
kernel project with a close-knit community of developers.  We should set
the right example for these developers by making sure we do the right thing
from the start.

The Future
----------
2.5.x is almost upon us.  Are we prepared for it yet?  Besides that, what
are some of our ideas in regards to providing better documentation and tool
support?  I think we should revisit the "roadmap" or "tasklist" and (if
you have the time to be able to) start writing down deadlines for the
things we want done.  I didn't touch documentation at all in this RFC (or
the lack thereof :), or some of the problems with using GNU tools with
LinuxSH, hopefully we can see people getting more proactive around this.

Conclusion
==========

Besides the issues I've raised, I've always been impressed with the rapid
progress of LinuxSH.  The fact that NIIBE was able to quintiple the speed
of his board within 3 kernel revisions is just amazing :).  I think that if
we start making the effort to write better code and establish a consistent
interface, we can make even better progress.  New SH developers looking
to port LinuxSH to their favorite board will have a much easier time doing
so.  Patch review will allow us to ensure that the best code goes in, and
will help minimize conflicts.  Overall, we'll have a much better port.

Comments?

Marcus