|
From: Anton A. <ai...@ca...> - 2004-05-07 21:31:03
|
Hi all, I just sent of this new NTFS release 2.1.8 to Andrew and Linus for inclusion into the next 2.6 kernel release. At this point I would also like to say that I am no longer going to work on NTFS on 2.4 kernels as many of the write support changes to come as well as all the NFS changes from the 2.1.7 release are 2.6 kernel specific and backporting them to 2.4 kernels would take far too much time which I do not have. So if you want to keep up with the latest NTFS driver, I am afraid you will have to switch to a 2.6 kernel. Of course, other people (Pawel (-;) please feel free to provide backports if you wish... Here is the email I sent to Andrew and Linus: Hi Andrew, Hi Linus, please do a bk pull http://linux-ntfs.bkbits.net/ntfs-2.6 Thanks! This is the next NTFS release 2.1.8. It goes a long way towards ensuring the safety of people's data by refusing read-write mounts when the inode mirror or the journal are dirty and by emptying the journal when it is clean so Windows cannot cause data corruption by replaying a stale journal. Also, the new nano-second kernel time precision is used to improve on the NTFS <-> Linux time conversion functions. The detailed ChangeLog is in the descriptions of the ChangeSets below. Note the patch is rather large because of a lot of white space cleanups that happened while editing (removal of trailing spaces and conversion of spaces to tabs). This is tested with NTFS volumes with clean and dirty journals created by Win2k and XP and I am quite confident it will work with NT4 and Win2003 and later. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ =================================================================== This will update the following files: fs/ntfs/time.c | 82 ---- Documentation/filesystems/ntfs.txt | 13 fs/ntfs/ChangeLog | 46 ++ fs/ntfs/Makefile | 18 fs/ntfs/aops.c | 46 +- fs/ntfs/attrib.c | 39 + fs/ntfs/compress.c | 18 fs/ntfs/dir.c | 98 ++-- fs/ntfs/inode.c | 85 ---- fs/ntfs/inode.h | 53 ++ fs/ntfs/layout.h | 385 ++++++++++--------- fs/ntfs/logfile.c | 737 +++++++++++++++++++++++++++++++++++++ fs/ntfs/logfile.h | 305 +++++++++++++++ fs/ntfs/malloc.h | 11 fs/ntfs/mft.h | 15 fs/ntfs/mst.c | 56 +- fs/ntfs/ntfs.h | 22 - fs/ntfs/super.c | 520 ++++++++++++++++++++------ fs/ntfs/time.h | 100 +++++ fs/ntfs/types.h | 32 - fs/ntfs/volume.h | 26 - 21 files changed, 2070 insertions(+), 637 deletions(-) through these ChangeSets: <ai...@ca...> (04/04/24 1.1371.748.5) NTFS: Use get_bh() instead of manual atomic_inc() in fs/ntfs/compress.c. <ai...@ca...> (04/04/24 1.1371.748.6) NTFS: - Modify fs/ntfs/time.c::ntfs2utc(), get_current_ntfs_time(), and utc2ntfs() to work with struct timespec instead of time_t on the Linux UTC time side thus preserving the full precision of the NTFS time and only loosing up to 99 nano-seconds in the Linux UTC time. - Move fs/ntfs/time.c to fs/ntfs/time.h and make the time functions static inline. <ai...@ca...> (04/04/25 1.1371.748.7) NTFS: - Remove unused ntfs_dirty_inode(). - Cleanup super operations declaration. <ai...@ca...> (04/04/25 1.1371.748.8) NTFS: Wrap flush_dcache_mft_record_page() in #ifdef NTFS_RW. <ai...@ca...> (04/04/25 1.1371.748.9) NTFS: Add NInoTestSetFoo() and NInoTestClearFoo() macro magic to fs/ntfs/inode.h and use it to declare NInoTest{Set,Clear}Dirty. <ai...@ca...> (04/04/25 1.1371.748.10) NTFS: Move typedefs for ntfs_attr and test_t from fs/ntfs/inode.c to fs/ntfs/inode.h so they can be used elsewhere. <ai...@ca...> (04/04/25 1.1371.748.11) NTFS: Determine the mft mirror size as the number of mirrored mft records and store it in ntfs_volume->mftmirr_size (fs/ntfs/super.c). <ai...@ca...> (04/04/27 1.1525.1.3) NTFS: Load the mft mirror at mount time and compare the mft records stored in it to the ones in the mft (fs/ntfs/super.c). <ai...@ca...> (04/04/28 1.1589) NTFS: - Fix compiler warnings related to type casting. - Move %L to %ll as %L is floating point and %ll is integer which is what we want. - Add logfile inode to ntfs_volume structure and the code to clean it up in super.c. <ai...@ca...> (04/05/07 1.1594) NTFS: Read the journal ($LogFile) and determine if the volume has been shutdown cleanly and force a read-only mount if not (fs/ntfs/super.c and fs/ntfs/logfile.c). This is a little bit of a crude check in that we only look at the restart areas and not at the actual log records so that there will be a very small number of cases where we think that a volume is dirty when in fact it is clean. This should only affect volumes that have not been shutdown cleanly and did not have any pending, non-check-pointed i/o. <ai...@ca...> (04/05/07 1.1595) NTFS: Eeek. Forgot to revert the Makefile before checking it in last time... <ai...@ca...> (04/05/07 1.1597) NTFS: 2.1.8 release - If the $LogFile indicates a clean shutdown and a read-write (re)mount is requested, empty $LogFile by overwriting it with 0xff bytes to ensure that Windows cannot cause data corruption by replaying a stale journal after Linux has written to the volume. =================================================================== This is the simple GNU diff -urNp for non-BK users: I am not including the patch as it is far too big (165kiB) but you can find it here if you want it: http://www-uxsup.csx.cam.ac.uk/~aia21/linux/ The file is linux-2.6.6-rc3-bk-ntfs-2.1.8.diff or if you prefer a smaller file, linux-2.6.6-rc3-bk-ntfs-2.1.8.diff.bz2. |
|
From: Szakacsits S. <sz...@si...> - 2004-05-07 21:49:50
|
On Fri, 7 May 2004, Anton Altaparmakov wrote: > At this point I would also like to say that I am no longer going to work > on NTFS on 2.4 kernels as many of the write support changes to come What's the plan, priorities, schedule? Szaka |
|
From: Anton A. <ai...@ca...> - 2004-05-08 10:05:11
|
On Fri, 7 May 2004, Szakacsits Szabolcs wrote: > On Fri, 7 May 2004, Anton Altaparmakov wrote: > > > At this point I would also like to say that I am no longer going to work > > on NTFS on 2.4 kernels as many of the write support changes to come > > What's the plan, priorities, schedule? plan: Slowly and extremely carefully move the code from ntfs-2.5-devel repository to the ntfs-2.6 repository splitting it up into obviously correct chunks, one release each. ntfs-2.5-devel has a nasty deadlock in it which I never had time to debug and last time I tried to debug it I didn't manage to trigger the deadlock so it is possible that the deadlock was in the generic kernel rather than NTFS, I don't know. But to make sure ntfs-2.6 doesn't end up with a deadlock I am going to do everything in very small obviously correct chunks and release each of them so that they will get maximum testing and should a deadlock occur despite me taking care not to have one than it should be relatively easy to start backing out releases one by one until the deadlock goes away. (Or this is the idea anyway...) The last two releases had a lot of changes but that was because the changes were mostly trivial cleanups or addition of easily understandable/obviously correct code that cannot cause deadlocks. Apart from that there isn't much of a plan, really, except the same what I have always said about how I want to go about implementing write support and I don't want to repeat myself now. (Mostly because I can't remember... (-;) priorities: See above plan. schedule: No such thing... We got some small donations and so I spent quite a lot of time working on the driver as they both asked for "quicker ntfs write support" and I have gotten the driver a little closer to that with the last two releases. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Szakacsits S. <sz...@si...> - 2004-05-08 13:20:51
|
On Sat, 8 May 2004, Anton Altaparmakov wrote: > On Fri, 7 May 2004, Szakacsits Szabolcs wrote: > > On Fri, 7 May 2004, Anton Altaparmakov wrote: > > > > > At this point I would also like to say that I am no longer going to work > > > on NTFS on 2.4 kernels as many of the write support changes to come > > > > What's the plan, priorities, schedule? > > plan: Slowly and extremely carefully move the code from ntfs-2.5-devel > repository to the ntfs-2.6 repository splitting it up into obviously > correct chunks, one release each. I see: 2.2.0-WIP - More write support: Add inode writing and resident file overwrite. http://linux-ntfs.bkbits.net:8080/ntfs-2.5-devel/anno/fs/ntfs/ChangeLog@1.157?nav=index.html|src/|src/fs|src/fs/ntfs|hist/fs/ntfs/ChangeLog|diffs/fs/ntfs/ChangeLog@1.157 Were these done recently or BK has some datum bugs? It says these 2.2.0-WIP changes were done in 2002 :-o > Apart from that there isn't much of a plan, really, except the same > what I have always said about how I want to go about implementing > write support and I don't want to repeat myself now. No need to repeat yourself, it would be more than enough if you pointed out where you wrote about it once. Unfortunately I've never seen this written on this developer list in the last two years, also nothing such thing on the web site. Even google failed for me. Or is it the TODO part in the driver's ChangeLog? BK also says it's from 2002. Is that correct or a BK misfunction? But please don't feel pressured to answer, people just asking me too and I don't have the slightest clue about your plans, I can give just optimistic estimates how long a full or better partial NTFS write support can take if the project restarted the work on it. Your original sentences suggested for me that significant changes, new big features are coming soon what apparently I misunderstood. Cheers, Szaka |
|
From: Anton A. <ai...@ca...> - 2004-05-08 17:20:54
|
On Sat, 8 May 2004, Szakacsits Szabolcs wrote: > On Sat, 8 May 2004, Anton Altaparmakov wrote: > > On Fri, 7 May 2004, Szakacsits Szabolcs wrote: > > > On Fri, 7 May 2004, Anton Altaparmakov wrote: > > > > At this point I would also like to say that I am no longer going to work > > > > on NTFS on 2.4 kernels as many of the write support changes to come > > > > > > What's the plan, priorities, schedule? > > > > plan: Slowly and extremely carefully move the code from ntfs-2.5-devel > > repository to the ntfs-2.6 repository splitting it up into obviously > > correct chunks, one release each. > > I see: > 2.2.0-WIP - More write support: Add inode writing and resident file overwrite. > > http://linux-ntfs.bkbits.net:8080/ntfs-2.5-devel/anno/fs/ntfs/ChangeLog@1.157?nav=index.html|src/|src/fs|src/fs/ntfs|hist/fs/ntfs/ChangeLog|diffs/fs/ntfs/ChangeLog@1.157 > > Were these done recently or BK has some datum bugs? It says these > 2.2.0-WIP changes were done in 2002 :-o Correct! It was a looong time ago. I never had time since to work on it really... and since it had the deadlock I couldn't just merge/release it so it has been dormant since then. Only recently I got to looking at it again and noticed that a lot of the bits were quite trivial and I added the logfile checking / dirty detection and that was what went into 2.1.7 and 2.1.8. > > Apart from that there isn't much of a plan, really, except the same > > what I have always said about how I want to go about implementing > > write support and I don't want to repeat myself now. > > No need to repeat yourself, it would be more than enough if you pointed > out where you wrote about it once. Unfortunately I've never seen this > written on this developer list in the last two years, also nothing such > thing on the web site. Even google failed for me. > > Or is it the TODO part in the driver's ChangeLog? BK also says it's from > 2002. Is that correct or a BK misfunction? That is that old too... I am afraid it is quite possible that my plans have only ever existed on IRC and/or private emails... > But please don't feel pressured to answer, people just asking me too and I > don't have the slightest clue about your plans, I can give just optimistic > estimates how long a full or better partial NTFS write support can take if > the project restarted the work on it. > > Your original sentences suggested for me that significant changes, new big > features are coming soon what apparently I misunderstood. Not that soon, no. Although having resident file overwrite is a pretty important feature. And being able to write mft records / dirty inodes is a pretty big feature for the driver internals as it allows all sorts of code to be added in various places to do different things. E.g. allows to start on file truncation support which means you can write and resize files. It also allows work on access time updating to start. And if anyone is so inclined (FlatCap was working on this but he is deNTFSing his life at the moment...) directory writing code can start to be added. All sorts of things can start happenning really... I will get back to you at a later time with what my personal plans for working on the driver look like. At the moment I have my 8-month-old girl sitting on my lap and I need to go and play with her as she is getting rather bored... Cheers, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Szakacsits S. <sz...@si...> - 2004-05-08 20:32:23
|
On Sat, 8 May 2004, Anton Altaparmakov wrote: > On Sat, 8 May 2004, Szakacsits Szabolcs wrote: > > > Were these done recently or BK has some datum bugs? It says these > > 2.2.0-WIP changes were done in 2002 :-o > > Correct! It was a looong time ago. I never had time since to work on it > really... and since it had the deadlock I couldn't just merge/release it > so it has been dormant since then. I wasn't aware there are more code, otherwise probably I would have checked it out for the problems ... Do you remember when over a year ago we debugged and fixed a deadlock what people sometimes reported? http://archives.mandrakelinux.com/cooker/2003-02/msg00285.php http://archives.mandrakelinux.com/cooker/2003-02/msg00538.php Isn't it possible you still mean this bug, just forgot about it that's why you can't reproduce anymore? > I am afraid it is quite possible that my plans have only ever existed > on IRC and/or private emails... Potential sponsors would be interested also, I'm quite sure ;-) > > Your original sentences suggested for me that significant changes, new big > > features are coming soon what apparently I misunderstood. > > Not that soon, no. Although having resident file overwrite is a pretty > important feature. I totally agree. Probably I wasn't clear either, I do know a lot of work is needed still and even something that looks a trival feature or non-feature can take a long time to get done properly by careful, hard work. I just thought you became a beneficent millionaire or someone seriously started to sponsor the development ;-) > And being able to write mft records / dirty inodes is a pretty big > feature for the driver internals as it allows all sorts of code to be > added in various places to do different things. Of course I agree. I wasn't aware this code already exists in the development tree, I thought it's the same as the one in the kernel so I've never checked it out. Anyway, the above is a very good news. > E.g. allows to start on file truncation support which means you can > write and resize files. Yes, create, unlink, truncate would be the most important next steps, I think. > It also allows work on access time updating to start. And if anyone > is so inclined (FlatCap was working on this but he is deNTFSing his > life at the moment...) directory writing code can start to be added. > All sorts of things can start happenning really... Anybody is working on anything related? Either user space or kernel space? I'm aware of your ntfstruncate but nothing else. > I will get back to you at a later time with what my personal plans for > working on the driver look like. Great. These should be summarzed on the web also somehow IMHO. Something that what and how people could help. Not necessarily kernel coding but also updating, cleaning documantation, web pages, testing, etc. There would be many ways to help out. Cheers, Szaka |
|
From: Anton A. <ai...@ca...> - 2004-05-09 20:56:00
|
On Sat, 8 May 2004, Szakacsits Szabolcs wrote: > On Sat, 8 May 2004, Anton Altaparmakov wrote: > > On Sat, 8 May 2004, Szakacsits Szabolcs wrote: > Do you remember when over a year ago we debugged and fixed a deadlock what > people sometimes reported? > > http://archives.mandrakelinux.com/cooker/2003-02/msg00285.php > http://archives.mandrakelinux.com/cooker/2003-02/msg00538.php I don't remember to be honest and I cannot open archives.mandrakelinux.com a all. )-: > Isn't it possible you still mean this bug, just forgot about it that's why > you can't reproduce anymore? That depends what it was but I very much doubt it as when the deadlock appeared it was just when I had implemented the inode write out in the VM page writeback (writepage()) code path. The code deadlocked always after writing something after a few seconds (the VM tried to write out the dirty pages). I then got very busy and development stopped. I revisited the code several months later and the deadlock was still occuring but not always. I then got busy again and only went back to trying it recently (a few months ago now I think) and now it didn't deadlock again which makes me wonder if something in the kernel changed and/or got fixed and that was the cause for the deadlocks... I don't know but I would rather not just take the code from ntfs-2.5-devel into ntfs-2.6 to be on the safe side. Also, I have had some ideas as to how to improve on the situation due to VM improvements. My ideas should remove quite a lot of code from the one in ntfs-2.5-devel and they should remove any eventual deadlocks, too. but we will see... > > I am afraid it is quite possible that my plans have only ever existed > > on IRC and/or private emails... > > Potential sponsors would be interested also, I'm quite sure ;-) (-: > > > Your original sentences suggested for me that significant changes, new big > > > features are coming soon what apparently I misunderstood. > > > > Not that soon, no. Although having resident file overwrite is a pretty > > important feature. > > I totally agree. > > Probably I wasn't clear either, I do know a lot of work is needed still > and even something that looks a trival feature or non-feature can take a > long time to get done properly by careful, hard work. I just thought you > became a beneficent millionaire or someone seriously started to sponsor > the development ;-) I wish I had become a millionaire! (-: I found devote myself to Linux fulltime on the spot... (-: And yes, proper funding for NTFS development would be rather nice... > > And being able to write mft records / dirty inodes is a pretty big > > feature for the driver internals as it allows all sorts of code to be > > added in various places to do different things. > > Of course I agree. I wasn't aware this code already exists in the > development tree, I thought it's the same as the one in the kernel so > I've never checked it out. > > Anyway, the above is a very good news. > > > E.g. allows to start on file truncation support which means you can > > write and resize files. > > Yes, create, unlink, truncate would be the most important next steps, > I think. > > > It also allows work on access time updating to start. And if anyone > > is so inclined (FlatCap was working on this but he is deNTFSing his > > life at the moment...) directory writing code can start to be added. > > All sorts of things can start happenning really... > > Anybody is working on anything related? Either user space or kernel space? > I'm aware of your ntfstruncate but nothing else. FlatCap did a lot of work on understanding how the B-trees (apparently a B*tree and not a B+tree as Microsoft claim). I am not too sure about this stuff. You would need to ask him. I know he has some stuff in the tng-support BK repository relating to the Btree business... > > I will get back to you at a later time with what my personal plans for > > working on the driver look like. > > Great. These should be summarzed on the web also somehow IMHO. Something > that what and how people could help. Not necessarily kernel coding but > also updating, cleaning documantation, web pages, testing, etc. There > would be many ways to help out. I will try and come up with something, maybe tomorrow, in email form and if worst comes to worst I/someone can always just cut and paste the email onto our webpages in the Status page or something... Don't know if I will have time as I am rather busy at the moment and I also need to make new releases for both the driver and ntfsprogs with the fixes for the decompression code I did this weekend. Two bugs in driver's compress.c and three in libntfs' compress.c that showed up by the bug report from Marcin Gibula (I know the l is not supposed to be an l but I don't think pine will allow me to write the correct character). I really also ought to run the old compress.c and the old compress.c though MD5 summing lots of compressed files to make sure I haven't broken anything with my fixes but I am not sure I will have time for that. I might just release it and wait for people to complain. My driver fixes certainly made things work ok for Marcin. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Szakacsits S. <sz...@si...> - 2004-05-09 21:39:28
|
On Sun, 9 May 2004, Anton Altaparmakov wrote: > On Sat, 8 May 2004, Szakacsits Szabolcs wrote: > > Do you remember when over a year ago we debugged and fixed a deadlock what > > people sometimes reported? > > > > http://archives.mandrakelinux.com/cooker/2003-02/msg00285.php > > http://archives.mandrakelinux.com/cooker/2003-02/msg00538.php > > I don't remember to be honest and I cannot open archives.mandrakelinux.com > a all. )-: Apparently it's down now. That deadlock was fixed in 2.1.1 but based on your later bug description it doesn't sounds anymore the two deadlocks would have been the same. A different site for that old thread, http://marc.theaimsgroup.com/?l=linux-ntfs-dev&m=104602041214371&w=2 Later, Szaka |
|
From: Richard R. <nt...@fl...> - 2004-05-11 21:32:27
|
Hi guys, > > > (FlatCap ... is deNTFSing his life at the moment...) Yep. I've been ignoring pleas for help and neglecting the website. I feel much more relaxed now :-) > > Yes, create, unlink, truncate would be the most important next steps, > > I think. Sounds good to me. > > > It also allows work on access time updating to start. And if anyone > > > is so inclined directory writing code can start to be added. Over the last week, I've been trying to turn my directory notes into code. There's not really anything to see yet - just some hacked up userspace code to represent the index root and alloc. What state is the code to manipulate MFT records? I'd like to be able to add a record, e.g. INDEX_ALLOC, change the size of a record, e.g. INDEX_ROOT and be able to ask how much space is free (in the record). > FlatCap did a lot of work on understanding how the B-trees. If anyone's interested, I wrote up my notes ages ago (with pictures): http://linux-ntfs.sourceforge.net/ntfs/concepts/tree/index.html See the add/del pages, too. > You would need to ask him. I know he has some stuff in the > tng-support BK repository relating to the Btree business... That code shows how to add/del items from a b*tree (the sort of tree that ntfs uses). > > Great. These should be summarized on the web also somehow IMHO. We need more than that, much more. IMHO, the news needs to be less formal, and needs tell people how we're getting on. At the moment we have quite a lots of page hits, but I doubt anyone comes back! > Something about what, and how, people could help. We keep scaring people off. Beginners don't want to do the boring tasks and they can't manage the sexy stuff. > Not necessarily kernel coding but also updating, cleaning > documentation, web pages, testing, etc. Yes, great. And I think that replacing the help forums with a wiki would be a step forward. Get a community rather than a few individuals. But... who's going to _manage_ all that. It won't just happen on its own. [sigh] tools are useful, but people want write support we have the skills to deliver write support it's almost impossible to deliver write support in a piecemeal fashion ntfsprogs / website distract us from development suggestions? (in no particular order) stop ntfsprogs devel campaign for money (to work full time) close the mailing lists give priority to driver a big PR campaign recruitment drive campaign for developers (to spread the load) give up close the website give priority to ntfsprogs FlatCap (Rich) |
|
From: Szakacsits S. <sz...@si...> - 2004-05-12 02:32:51
|
On Tue, 11 May 2004, Richard Russon wrote: > Hi guys, Hi, nice to see you again :) > > > > It also allows work on access time updating to start. And if anyone > > > > is so inclined directory writing code can start to be added. > > Over the last week, I've been trying to turn my directory notes into > code. There's not really anything to see yet - just some hacked up > userspace code to represent the index root and alloc. Sounds cool! > What state is the code to manipulate MFT records? I'd like to be able > to add a record, e.g. INDEX_ALLOC, change the size of a record, e.g. > INDEX_ROOT and be able to ask how much space is free (in the record). Anton could comment in detail. I think there is no add/delete but IMHO they shouldn't be difficult unless AT_ATTRIBUTE_LIST is needed. AFAIK resize is close to OK. [the incomplete] ntfstruncate needed these. > If anyone's interested, I wrote up my notes ages ago (with pictures): > http://linux-ntfs.sourceforge.net/ntfs/concepts/tree/index.html > See the add/del pages, too. Thanks, I'm sure people are interested ;-) I'll definitely check it out. > IMHO, the news needs to be less formal, What do you exactly mean? IMHO some problem with the news are, - sometimes it's misssing items (see e.g. the recent 2.4 and 2.6 driver releases) - usually it's about only software releases, except when you were adding other news (e.g. FAQ translations) - the 4 items list is too short, there could be a bit more and maybe old history > and needs tell people how we're getting on. Definitely. And perhaps who is working on different parts (mostly nobody). My impression is that people are not aware how huge is this task. IMHO NTFS could be compared only with XFS or Reiser4. Both are worked on by 6-8 people, full time. NTFS moreover also undocumented, officially - and some parts still even unofficially ... > At the moment we have quite a lots of page hits, 330,000 a month and soon 5 million total. > We keep scaring people off. How? If we know we could try to address it. > Beginners don't want to do the boring tasks and they can't manage the > sexy stuff. If we explained why a "boring task" is important and actually it is part of a sexy stuff then it maybe wouldn't look so boring anymore. We chould lists many tasks, explain why they are important and let people choose what they prefer. However I'm aware that even preparing such a list isn't trivial. So the first task could be: make a tasks list. > > Not necessarily kernel coding but also updating, cleaning > > documentation, web pages, testing, etc. > Yes, great. And I think that replacing the help forums with a wiki > would be a step forward. The wiki sounds a very good idea! But I think an adequate forum is also needed. I dislike them (though they are still better than Anton's unreadable quoted-printable emails ;) but most people who need help apparently like it. Far the most popular seems to be http://www.phpbb.com/ I don't know if it can be [two-way] gatewayed with mailing lists. > Get a community rather than a few individuals. But... who's going to > _manage_ all that. It won't just happen on its own. It won't. First we would need to know what exactly needs to be managed and either somebody (or more people) accepts it or it goes to the task list for potential helpers in the future. > tools are useful, but people want write support Some tools are needed to "support" write support: mkntfs, ntfsck, ntfsdb (debugger, editor), test suite. We have mkntfs. I'm close to convert and rip off the consistency check part from ntfsresize to reusable functions (for ntfsck, ntfsclone, etc). But no ntfsdb (ntfsinfo could develop to that direction?) and test suite (mine is totally ntfsresize focused). > suggestions? (in no particular order) > > stop ntfsprogs devel I think the above tools are needed. ntfsprogs is also an easy and "safe" playground, as Anton is wont to say. > campaign for money (to work full time) IMHO this needs a serious project plan or good luck. > close the mailing lists ??? > give priority to driver I think it has priority. > a big PR campaign > recruitment drive > campaign for developers (to spread the load) Yes but perhaps not before a detailed project plan. > give up Sorry, this one is exceptionally rejected :) Cheers, Szaka |
|
From: Anton A. <ai...@ca...> - 2004-05-12 22:26:14
|
Argh! pine just lost the email I wrote so I am going to try and piece it back together... On Wed, 12 May 2004, Szakacsits Szabolcs wrote: > My impression is that people are not aware how huge is this task. IMHO > NTFS could be compared only with XFS or Reiser4. Both are worked on by 6-8 > people, full time. NTFS moreover also undocumented, officially - and some > parts still even unofficially ... Indeed. NTFS is just a monster FS and people don't really appreciate/know this. > How? If we know we could try to address it. > > > Beginners don't want to do the boring tasks and they can't manage the > > sexy stuff. > > If we explained why a "boring task" is important and actually it is part > of a sexy stuff then it maybe wouldn't look so boring anymore. > > We chould lists many tasks, explain why they are important and let people > choose what they prefer. However I'm aware that even preparing such a list > isn't trivial. So the first task could be: make a tasks list. Ah, but do you really want to sacrifice your development time to do that? It would take quite a lot of effort to set this up and then to keep it always up to date, etc... Time has always been the killer problem for us all. We don't have much/any of it spare so doing anything more than a bit of development is kind of not an option or the development stops altogether. Just as an example take this evening for me. I have answered five-ten emails (some non-NTFS related) and it is pretty much bed time for me as I have to get up early for work and the result is that I haven't been able to write a single line of code this evening. The last few weeks I have done the opposite: ignored emails mostly and done programming each evening and I got a lot done. Unfortunately I have so little time that I cannot do both programming and keeping in touch with everything/updating website, etc. )-: And I think this is a common problem among all of us. > > > Not necessarily kernel coding but also updating, cleaning > > > documentation, web pages, testing, etc. > > Yes, great. And I think that replacing the help forums with a wiki > > would be a step forward. > > The wiki sounds a very good idea! Yes, that would be quite cool. > But I think an adequate forum is also needed. I dislike them (though they > are still better than Anton's unreadable quoted-printable emails ;) but Yes, sorry. I am using pine at the moment so they should be ok. (-: If anyone knows how to make evolution not use quoted printable encoding please let me know... > most people who need help apparently like it. Far the most popular seems > to be http://www.phpbb.com/ I don't know if it can be [two-way] gatewayed > with mailing lists. Ah, but do we care about people who need help? I would say ignore them all and close all the forums so they don't bother us anymore and no longer take away all our development time. We do not need to support NTFS users. No one is paying us for it... Maybe we could charge for answering support questions? (-; > > Get a community rather than a few individuals. But... who's going to > > _manage_ all that. It won't just happen on its own. > > It won't. First we would need to know what exactly needs to be > managed and either somebody (or more people) accepts it or it > goes to the task list for potential helpers in the future. Inded. But the problems is we need someone to maintain/manage that, too, i.e. the task list etc. And there is noone to do it. Catch 22! > > tools are useful, but people want write support > > Some tools are needed to "support" write support: mkntfs, ntfsck, ntfsdb > (debugger, editor), test suite. > > We have mkntfs. I'm close to convert and rip off the consistency check > part from ntfsresize to reusable functions (for ntfsck, ntfsclone, etc). > But no ntfsdb (ntfsinfo could develop to that direction?) and test suite > (mine is totally ntfsresize focused). True. > > suggestions? (in no particular order) > > > > stop ntfsprogs devel > > I think the above tools are needed. ntfsprogs is also an easy and "safe" > playground, as Anton is wont to say. Yes, and I did. (-: > > give priority to driver > > I think it has priority. Agreed. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Richard R. <nt...@fl...> - 2004-05-19 00:11:33
|
Hi, > > IMHO, the news needs to be less formal, > What do you exactly mean? We only create news for releases. We almost need some more social, more human, news. I have no idea what effect it would have :-) > IMHO some problem with the news are, > - sometimes it's missing items (see e.g. the recent 2.4 and 2.6 > driver releases) Hmm... that needs to be addressed. The news also needs to be mirrored to FreshMeat. > - the 4 items list is too short, there could be a bit more and > maybe old history OK, I've just expanded it to 8 items. It's just a parameter in the build script in the parts directory. If I get bored I might add more news items in summary only. > > and needs tell people how we're getting on. > Definitely. And perhaps who is working on different parts (mostly nobody). This might be where things break down. For example, I started working on the directory handling *years* ago, but only now do I have some code. I would like to tell the world what I'm doing, but I don't want to feel *obliged* to deliver on some timescale. We all have more important (non-NTFS) commitments to fulfil. > My impression is that people are not aware how huge is this task. IMHO > NTFS could be compared only with XFS or Reiser4. Both are worked on by 6-8 > people, full time Given that NTFS is a dirty word amongst some Linux distros, we cannot possibly compete. > > At the moment we have quite a lots of page hits, > 330,000 a month and soon 5 million total. Great isn't it :-) but how do we harness it? > > We keep scaring people off. > How? If we know we could try to address it. You've already answered that: NTFS is too huge. I've answered a lot of emails from people wanting to help, but few ever respond. I've probably been aiming too high. > > Beginners don't want to do the boring tasks and they can't manage the > > sexy stuff. > If we explained why a "boring task" is important and actually it is part > of a sexy stuff then it maybe wouldn't look so boring anymore. Suddenly this sounds like lots of effort (in management) for very little return. We're all techies; we're not managers. If we're not coding we're not using out time efficiently. > > tools are useful, but people want write support > Some tools are needed to "support" write support: mkntfs, ntfsck, ntfsdb > (debugger, editor), test suite. So perhaps we need to stop doing "useful" tools and only create development aids. > > suggestions? (in no particular order) > > stop ntfsprogs devel > I think the above tools are needed. ntfsprogs is also an easy and "safe" > playground, as Anton is wont to say. That's fair enough. I've started my directory code in userspace for the simplicity. I need to get my UML debug environment working again. > > campaign for money (to work full time) > IMHO this needs a serious project plan or good luck. Yeah, but it's worth imagining. To raise a few thousand probably that big a deal - just enough for a few months sabbatical. (I have a job and I don't want to quit quite yet). > > close the mailing lists > ??? Just every random thing that popped into my head. Perhaps you could replace that with "make the list dev only (i.e. no more help)" I like helping people, but I should be coding. > > give priority to driver > I think it has priority. I don't think it does, atm. We're all busy with real life, so development is a fix here, or a quick utility there. We've all been too daunted to start the next stage. Hell, I've been hiding from it for longer than both of you. > > give up > Sorry, this one is exceptionally rejected :) Glad to hear it :-) Rich. |
|
From: Szakacsits S. <sz...@si...> - 2004-05-19 01:27:05
|
Hi, On Wed, 19 May 2004, Richard Russon wrote: > > > IMHO, the news needs to be less formal, > > What do you exactly mean? > We only create news for releases. We almost need some more social, more > human, news. I have no idea what effect it would have :-) You mean something like this? ;) http://linux-ntfs.sourceforge.net/info/ntfsresize.html May 17, 2004 WARNING! Be careful when partitioning on Linux 2.6 kernels! Some combinations of certain BIOS settings and certain partitioning softwares running on Linux 2.6 kernels MIGHT create incorrect partition table that breaks the Windows boot process. If your NTFS was resized at the same time then your data is still undamaged. The Windows boot problem is recoverable and filesystem independent. Reports are from users of Mandrake 10, SUSE 9.1 and Fedora 2. Please see more details below. Of course almost all people, media blame ntfs resizing even if they had only FAT32 or the disk was already partitioned previously. > > IMHO some problem with the news are, > > - sometimes it's missing items (see e.g. the recent 2.4 and 2.6 > > driver releases) > > Hmm... that needs to be addressed. The news also needs to be mirrored > to FreshMeat. As we agreed long ago I did/do the FreshMeat stuff for ntfsprogs, the 2.4 and 2.6 drivers. > > - the 4 items list is too short, there could be a bit more and > > maybe old history > > OK, I've just expanded it to 8 items. It's just a parameter in the > build script in the parts directory. If I get bored I might add more > news items in summary only. Cool, I think it's indeed better :) > > > and needs tell people how we're getting on. > > Definitely. And perhaps who is working on different parts (mostly nobody). > This might be where things break down. For example, I started working > on the directory handling *years* ago, but only now do I have some > code. I would like to tell the world what I'm doing, but I don't want > to feel *obliged* to deliver on some timescale. We all have more > important (non-NTFS) commitments to fulfil. We'd just need to state it: "working on this if time permits, no time estimate, no obligation, feel free to ask". My "short" term plan (no estimate, no obligation, feel free to ask :) - finish ripping off some useful stuffs from ntfsresize to be reusable (consistency check, mount check, progress bar, ...) - rudimentary ntfsck (consistency check and whatever badly needed). - refactor ntfsclone a bit - try to quickly replace partimage's own, experimental ntfs code by the stable ntfsprogs (looks much more time saving then adding the features people ask for ntfsclone) - create file > > My impression is that people are not aware how huge is this task. IMHO > > NTFS could be compared only with XFS or Reiser4. Both are worked on by 6-8 > > people, full time > > Given that NTFS is a dirty word amongst some Linux distros, [...] Only Red Hat. > > > tools are useful, but people want write support > > Some tools are needed to "support" write support: mkntfs, ntfsck, ntfsdb > > (debugger, editor), test suite. > > So perhaps we need to stop doing "useful" tools and only create > development aids. I think so. But only those development aids what are needed and when needed. > > > campaign for money (to work full time) > > IMHO this needs a serious project plan or good luck. > > Yeah, but it's worth imagining. To raise a few thousand probably that > big a deal - just enough for a few months sabbatical. BTW, just the extended attributes support for reiser4 cost minimum $US 600,000. Cheers, Szaka |
|
From: Will G. <wi...@le...> - 2004-05-11 22:42:32
|
FYI, I've been lurking the lists for many months. > > Something about what, and how, people could help. > > We keep scaring people off. Beginners don't want to do the boring tasks > and they can't manage the sexy stuff. True. I, for one, have a fairly good grasp of how NTFS works, in no small part due to the information published on the Linux-NTFS website. (I ended up writing an NTFS recovery toolset from scratch some time ago, which is how I ended up knowing what I do.) I have some amount of free time, and I'd like to see a write-capable NTFS driver, but I don't have any idea of where to start. The driver mystifies me, and there don't seem to be any resources describing its structure or status. From my perspective, the driver is a big fog. Knowing more about the kernel would probably help (doesn't it always), but even so. I think the lack of information -- or the low visibility of whatever exists -- is a serious problem, and major factor in "scaring away" prospective developers. --Will |
|
From: Anton A. <ai...@ca...> - 2004-05-12 22:39:11
|
Hi, On Tue, 11 May 2004, Will Glynn wrote: > FYI, I've been lurking the lists for many months. Nice to see you surfacing. (-: > > > Something about what, and how, people could help. > > > > We keep scaring people off. Beginners don't want to do the boring tasks > > and they can't manage the sexy stuff. > > True. I, for one, have a fairly good grasp of how NTFS works, in no small part > due to the information published on the Linux-NTFS website. (I ended up > writing an NTFS recovery toolset from scratch some time ago, which is how I > ended up knowing what I do.) I have some amount of free time, and I'd like to > see a write-capable NTFS driver, but I don't have any idea of where to start. > The driver mystifies me, and there don't seem to be any resources describing > its structure or status. > > From my perspective, the driver is a big fog. Knowing more about the kernel > would probably help (doesn't it always), but even so. I think the lack of > information -- or the low visibility of whatever exists -- is a serious > problem, and major factor in "scaring away" prospective developers. When I started writing the new driver there were a few good books on the kernel available online. I will see if I can find the URLs for them and see if they still exist / make sense with 2.6 kernels and email them to the list. They would definitely be required reading material to even stand the slightest chance of understanding the driver. Considering you have written tools I imagine you know C pretty well. That would be another prerequisite to doing kernel development. (C++ not allowed.) You also need to have understood and agreed with the kernel coding standards. Once one digests those one quickly gets used to them and then reading code (that adheres to them) is a LOT easier to do and to understand. I am now so familiar with it that I can just read some kernel code I have never seen before and immediately focus on what it is doing and understand it rather than having to stare at the looks and take a lot of time to extact the "what it actually does" from the "looks". A few years ago Rich and I spent about a week where I explained how the driver worked/was built up to him using IRC to talk to each other and a shared terminal windows (using the screen window manager) to show the code we were talking about. This was very successful but unfortunately we were not able to record it as the IRC log would not be too useful without the accompanying code display and highlighting at the same time... So I would need to find a lot of spare time to start writing up how the driver works... Maybe one day... For now I suggest anyone wanting to understand the driver to learn the kernel VFS layer first. The way to understand how a kernel FS works is to walk through exececuted code e.g.: system call open() -> sys_open() (in kernel) -> ... follow it through till you end up in ntfs code and follow that through, too. system call read() -> sys_read() (in kernel) -> ... as above. That way you can see how doing something from the user point of view gets into the NTFS code and what the NTFS code is expected to do by the VFS layer. The VFS layer does a LOT of the work for us so understanding it is really important to understand how the driver works... I will try and get those URLs in the next week or two. I am really busy at the moment and this weekend is my wife's birthday and I promised to make her a birthday cake so that will probably wipe out my weekend... (-; Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Anton A. <ai...@ca...> - 2004-05-12 22:09:04
|
Hi Rich, On Tue, 11 May 2004, Richard Russon wrote: > What state is the code to manipulate MFT records? I'd like to be able > to add a record, e.g. INDEX_ALLOC, change the size of a record, e.g. > INDEX_ROOT and be able to ask how much space is free (in the record). In kernel none of this is possible at present. In libntfs you can't yet add/remove records but that should be almost trivial (for the non-attribute list attribute present/necessary case). You can resize resident attributes both making them smaller and bigger and converting them to non-resident attributes if they support it. So you should be able to resize INDEX_ROOT without problems already. You just need to trerat EOPNOTSUP the same wasy ENOSPC. See libntfs/attrib.c::ntfs_resident_attr_resize(). For adding an attribute you would need to: Determine the amount of space needed by the new attribute. get search context ctx. ntfs_attr_lookup(...,ctx) for the attribute you want to insert and get errno == ENOENT. Then using the same ctx use a not yet existant function to create space in the mft record (see ntfsprogs/mkntfs.c::make_room_for_attribute()). If this returns ENOSPC then you know we can't do it. (In the future you will never get this unless you are asking for a too large chunk of resident attribute space (i.e. larger than an mft record) as we will be able to use extent records.) Then create the new attribute in the created space. and make it point properly at the next attribute. (See ntfsprogs/mkntfs.c::add_attr_index_alloc() and attr_attr_index_root() for examples and the functions they call.) Done. (-; Admittedly still a lot of work needed in the library to get all these functions in there but the end is in sight... > > > Great. These should be summarized on the web also somehow IMHO. > > We need more than that, much more. IMHO, the news needs to be less > formal, and needs tell people how we're getting on. At the moment we > have quite a lots of page hits, but I doubt anyone comes back! Probably true but who wants to write news items? > > Something about what, and how, people could help. > > We keep scaring people off. Beginners don't want to do the boring tasks > and they can't manage the sexy stuff. > > > Not necessarily kernel coding but also updating, cleaning > > documentation, web pages, testing, etc. > > Yes, great. And I think that replacing the help forums with a wiki > would be a step forward. Get a community rather than a few individuals. > But... who's going to _manage_ all that. It won't just happen on its > own. Yes wiki would probably be fantastic. But as you say all these things require people to do them and all of them would take away from our already scarce time for development... > [sigh] > > tools are useful, but people want write support > we have the skills to deliver write support > it's almost impossible to deliver write support in a piecemeal fashion > ntfsprogs / website distract us from development > > suggestions? (in no particular order) > > stop ntfsprogs devel No, we need that for testing/development I think. A lot easier than kernel even when using VMware or UML... > campaign for money (to work full time) But that takes a lot of time and effort in itself... > close the mailing lists Need them for bug reports and developer discussions... > give priority to driver Definitely. > a big PR campaign > recruitment drive > campaign for developers (to spread the load) All require a lot of time. )-: > give up No way! (-: We have come this far... > close the website Better to just say "This web site is no longer being updated due to lack of resources. If you want to help keep it uptodate please email us." at the top rather than shut it. You never know might get a few volunteers... (-; > give priority to ntfsprogs No, kernel is more important... At least to me... But utilities like ntfsresize, mkntfs, and ntfsclone are very useful for users and distros and to bring Linux to the masses of Windows users so we shouldn't just abandon them altogether... One added item: stop offering help to people; only answer bug reports and ignore everything else. Would save a _lot_ of time... (-; Cheers, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Richard R. <nt...@fl...> - 2004-05-19 00:12:20
|
Hi, > > What state is the code to manipulate MFT records? I'd like to be able > > to add a record, e.g. INDEX_ALLOC, change the size of a record, e.g. > > INDEX_ROOT and be able to ask how much space is free (in the record). > > In kernel none of this is possible at present. > ... (for the non-attribute list attribute present/necessary case)... hehe, the first write support is going to be funny. You can delete this file, *if* it's a leaf node and it's parent has 0xA0 and it has no 0x20 and it is resident and ... > For adding an attribute you would need to: > ... > Done. (-; You make it sound so easy. > Admittedly still a lot of work needed in the library to get > all these functions in there but the end is in sight... OK, cheers. There's no rush yet. > > We need more than that, much more. IMHO, the news needs to be less > > formal, and needs tell people how we're getting on. > Probably true but who wants to write news items? *sigh* Not me either. > > Yes, great. And I think that replacing the help forums with a wiki > Yes wiki would probably be fantastic. But as you say all these things > require people to do them and all of them would take away from our already > scarce time for development... > > suggestions? > All require a lot of time. )-: That sums it up well :-) > > give up > > No way! (-: We have come this far... Good answer. > > close the website > If you want to help keep it uptodate please email us. > You never know might get a few volunteers... Interesting. > One added item: > stop offering help to people; only answer bug reports and ignore > everything else. Would save a _lot_ of time... (-; Hmm, now that needs a separate email. Rich. |
|
From: Pawel K. <pk...@be...> - 2004-05-07 23:48:42
|
On Fri, 7 May 2004, Anton Altaparmakov wrote: > afraid you will have to switch to a 2.6 kernel. Of course, other people > (Pawel (-;) please feel free to provide backports if you wish... Well, I'll see what I can do... pkot -- mailto:pk...@be... http://www.gnokii.org/ |
