Project of the Month, September 2006

• Project name: Linux-NTFS
• Date founded/started: November 2000
• URL: www.linux-ntfs.org
• Project page: http://sourceforge.net/projects/linux-ntfs

Description of project

The goals of this project are to create a new Linux kernel driver for the NTFS file system (1.2, 3.x, and later), user space utilities (e.g. format, NTFS check, etc.), and a library to avoid code duplication and provide access to NTFS to other GPL programs.

On the driver side (kernel NTFS driver, ntfsmount, ntfs-3g), the project allows users to read and write to their NTFS partitions (the public kernel driver is mostly read-only at the moment whilst ntfsmount and ntfs-3g are read-write), thus allowing people to use their Windows files under Linux. On the utilities side, it allows people to create NTFS volumes (mkntfs), resize existing volumes (ntfsresize), create copies of volumes (for backup purposes for example — ntfsclone), and do all sorts of other things with NTFS volumes. On the library front, NTFS library (libntfs) provides API for other GPLed programs to access NTFS volumes, such as the recovery program TestDisk.

We are not quite unique in that there are other NTFS drivers out there, but all of them have drawbacks and are known to corrupt data, thus our solution is the only one that we trust with our and our users’ data.

Trove info

• Development Status: 3 – Alpha, 4 – Beta, 5 – Production/Stable
• Intended Audience: Developers, End Users/Desktop, System Administrators
• License: GNU General Public License (GPL)
• Operating System: All POSIX (Linux/BSD/UNIX-like OSes), Linux, Mac OS X
• Programming Language: C
• Topic: Filesystems, Linux
• Translations: English
• User Interface: Non-interactive (daemon)

Why and how did you get started?

Anton: I personally got started around 2000. I was using Linux and Windows dual-boot and wanted to share data between the two. At the time Linux had a basic read-only driver (the read-write support was known to be broken) but it was not maintained and it no longer compiled. The kernel maintainers wanted to throw it out. I stepped up to maintain it so I could keep using it. Out of this grew the Linux-NTFS project. I went on to develop the new NTFS driver (now present by default in 2.6 kernels and also present in a lot of distributions’ 2.4 kernels, including those of SUSE and Mandriva) and the ntfsprogs with their libntfs. I was joined by Richard Russon at first and later by Yura Pakhuchiy, Szabolcs Szakacsits, and Yuval Fledel, and several others who are now not active any more.

Yura: In June 2004 I was unhappy that I could not write to NTFS volumes from Linux. Eventually I decided to fix the problem myself.

Szaka: As the co-founder of the Hungarian Linux User Group over a decade ago, I had plenty of chances and possibilities to discuss Linux usage problems with users. Around in 2002 most talks stopped at the topic of how to install Linux non-destructively alongside Windows and how to read-write Windows data reliably with open source software. There was no solution, so I thought these issues need to be solved. I reviewed all Linux NTFS implementations, and this project was the only active one, so I contributed ntfsresize as my introduction. Then I worked on many things and finally released ntfs-3g beta in July, a full read-write NTFS driver, which is based on ntfsmount and uses the brilliant FUSE.

What is the software’s intended audience?

There are two audiences:

• Developers of installers and partitioning utilities can use our mkntfs code wrapped by their GUI applications to create NTFS volumes, and ntfsresize to resize NTFS volumes, for example. Developers of data rescue applications can use libntfs to manipulate NTFS volumes. Linux distribution developers can use the NTFS kernel driver or the userspace ntfsmount/ntfs-3g drivers to allow mounting of NTFS partitions.
• End users usually get our software because developers have integrated it with their programs and distributions, but for distributions such as Red Hat Enterprise Linux and Fedora Core, the management of which refuse to support NTFS altogether, users have to install NTFS by themselves, so they get to use it firsthand that way.

How many people do you believe are using your software?

Millions — perhaps even tens of millions. A lot probably don’t know it, because when they installed Linux on their Windows computer the installer used ntfsresize under the covers to shrink the Windows partition to make space for Linux. And then when they booted into Linux, their Windows partition was available under /windows/C. They may not know it, but it is thanks to our NTFS driver that this works. The tools are also used by FreeBSD guys and Cygwin, and we’ve had some reports of limited use in Mac OS X and Solaris.

What gave you an indication that your project was becoming successful?

Number of downloads and page views on SF.net and the number of support queries we get — all those have been rising. And we get incredibly positive feedback.

What has been your biggest surprise?

Anton: Perhaps how well the open source model works. I have been the only constant since I registered Linux-NTFS back in 2000 on SourceForge. I have seen several people come and go (and develop some cool stuff whilst they were here), but the project still remained and their work was picked up by people following them and improved and extended

Yuval: A Slashdot article that followed quickly after announcement of ntfs-3g on the developers mailing list.

Szaka: I sent the beta driver announcement to the NTFS developers mailing list for testing just before leaving for a trip. I was over the Arctic Circle when friends called to tell me that the ntfs-3g news was everywhere in the media all over the world.

What has been your biggest challenge?

Anton: Reverse-engineering NTFS to obtain enough information to be able to make this project happen and do safe write support to NTFS on Linux.

Yuval: Decrypting NTFS files that were encrypted with EFS in a non-standard way.

Yura: Some cases of attribute manipulation code and filenames collating (this mix of POSIX, WIN32, and DOS namespaces can really hurt the brain).

Szaka: The roots of NTFS stretch back more than 30 years, to the Files-11 filesystem of Digital Equipment Corp.’s PDP-11. Gary Kimura, one of the designers and implementors of NTFS, mentioned in one of his university lectures that the Windows NTFS codebase is 250,000 source lines. That’s about 50 man-years of work. We have to reverse-engineer that code, test, manage the project, and answer users’ questions. It’s a challenge to accomplish this well when you have limited free time.

Why do you think your project has been so well received?

Windows is the most widespread operating system, but a lot of people are trying out or switching to Linux. Our project makes it possible to install Linux alongside Windows and share the data.

Where do you see your project going?

Complete read/write support for NTFS on Linux, including fancy features such as writing to compressed files and reading/writing encrypted files, so you can install Linux on your NTFS drive. Stabilizing the unstable parts. Starting to focus on performance rather than conformance.

What’s on your project wish list?

Anton: Full read-write support; we are working on it. My personal wish list also includes an NTFS filesystem checker (a Windows chkdsk replacement) so people do not have to boot Windows to check/repair their NTFS volumes.

Yuval: Allowing Linux to use NTFS as the root/boot partition.

Yura: Ability to install Linux on an NTFS volume.

What are you most proud of?

Anton: The NTFS kernel driver, as that is pretty much written entirely by me and at the same time is the most challenging part of the project.

Yuval: Our wiki and our user support. Most forum questions get answered in a matter of hours. Developers answer faster than the original poster replies.

Yura: ntfsmount

If you could change something about the project, what would it be?

Yuval: I’d like to build a bigger developer community. The only people who answer question on the forums are the project developers. An open source project needs larger community support, not just larger deployment.

How do you coordinate the project?

Anton: Everyone can work on whatever they want. I make suggestions about what would be most helpful and people often work on that, but sometimes they prefer to work on something else instead. Seems to work quite well. Most discussion and bug reports take place on the linux-ntfs-dev list. Bugs are generally assigned depending on who wrote the code. Regression testing is spread out. Szaka has an automated test suite that he runs on several images of NTFS volumes. I personally use md5sum in the kernel driver to check that all files are still being read correctly (this was incredibly useful when working on the NTFS compression engine, for example).

Do you work on the project full-time, or do you have another job?

Anton: I work at the University of Cambridge (UK) at the Computing Service as a member of Unix Support group. I also work part time on NTFS for a computer company.

If you work on the project part-time, how much time would you say you spend, per week, on it?

Anton: At the moment, about 10-40 hours per week.

Yuval: A few hours per week.

Yura: Sometimes 20-30 hours per week, sometimes 0-5 hours.

Szaka: Daily maximum 1-2 hours, sometimes on the weekends a bit more.

What is your development environment like?

Anton: At work in Cambridge I use a Pentium 4 with 1G RAM, SUSE 10.1 (thus gcc4, etc). At home I use a Dual G5 with 512MB RAM and Mac OS X (again gcc4, etc), and an Athlon 1.33GHz, 1GB RAM, SUSE 10.0. I use IDA Pro, a powerful disassembler/debugger, to do most of my reverse engineering. Unfortunately its nice GUI works only on Windows, but it can debug Linux over the network. My text editor of choice is vim, and I use VMware extensively so I don’t have to reboot when doing testing.

Yuval: Windows XP, Cygwin, gcc, simple text editor, brain.

Yura: Athlon XP 2000 box with 1GB RAM, Gentoo Linux, vi, gcc, gdb, valgrind, lots of rxvt instances

Szaka: IBM T40, Linux, gcc, vi, slickedit, valgrind, xterms, qemu, and all sorts of filesystem test suites, benchmarks, and test utilities I’ve found or written.

Milestones:

• 2001: started work on new NTFS driver and on ntfsprogs/libntfs.
• April 2002 (kernel 2.5.11): the new kernel driver was added to the stock Linux kernel.
• May 2002 (kernel 2.4.18): the new driver was ported to 2.4.18 kernel by Pawel Kot with my help. Soon thereafter SUSE and others started adopting the new driver in their Linux distributions based on 2.4 kernels.
• June 2005 (ntfsprogs 1.10.0): a lot of code toward write support was added, but mostly not really used by programs.
• July 2005 (ntfsprogs 1.11.0): ntfsmount was added, really using the code from 1.10.0 release.
• October 2005 (ntfsprogs 1.12.0): limited file creation/deletion support was added to ntfsmount.
• Today most distributions bundle NTFS, both in terms of using ntfsresize to resize NTFS partitions at installation time and in terms of mounting NTFS volumes whilst running so people can access their files on NTFS volumes (most use the kernel NTFS driver but some are now thinking about including ntfs-3g).

Future Plans:

• Polishing code in CVS.
• Writing to compressed files and moving encrypted files access code to library.
• Some ideas about access permisions.
• ntfsprogs 2.0.0 (maybe 3.0.0?) release.
• Writing ntfsck, a filesystem checker for NTFS.

How can others contribute?

People can contribute in many ways. For example, we have Mario Emmenlauer who maintains our Web site/forums/wiki/new CVS server. Several people answer questions on the forums and mailing lists. Of course we welcome people who can program, too, but NTFS is a very complex file system and most people who have tried have soon left shaking their heads in despair.

The wiki contains a very good page on this subject with ideas in every field. If anyone is interested in anything, just drop a message to the developers mailing list. We will reply with specific ideas, and if that person will reply (most don’t), we can mentor him all the way. Don’t be afraid to join.