|
From: mark <mar...@ds...> - 2004-10-06 23:00:16
|
I trust I am directing this problem to the correct team. If I have made a
mistake, I apologise. I have tried to identify the correct addressee.
The problem I am experiencing was detected when I was making a Super Video CD
from a 5 minute video produced using Premiere under Windows NT4 SP6a. The
problem manifested itself as visual artefacts. I have traced the problem
back to the copying of a large *.avi file from an NTFS directory to an Ext3
folder. The size of the file is the same, but the content differs.
Repeating the test generates errors in different parts of the file. If I
were to take a look at the detailed "cmp" output I would suspect that buffer
management is not properly managed across the interrupt/realtime part and the
user priority part... It looks as though a block has been added or removed (I
cannot fathom which).
I don't know how to proceed analysing the problem any further.
Below I provide the configuration info the NTFS FAQ sect 3.4 asked for; the
console listing of an experiment I performed to localise the problem to the
NTFS filesystem; and the dmesg output to provide low level detail on hardware
config and software config; and a printout of /etc/fstab which shows the NTFS
options in force.
Please ask for further information if you would like it, but you might have to
tell me how to get it (i.e. which instructions to use).
Kind regards
Mark Atkinson
-----------------oOo----------------------
The NTFS FAQ asks for the following info taken from the text of the dmesg
response.
NTFS driver 2.1.6 [Flags: R/O MODULE].
NTFS volume version 1.2.
Linux version 2.6.3-16mdk (qa...@up...) (gcc version 3.3.2
(Mandrake Linux 10.0 3.3.2-6mdk)) #1 Fri Aug 13 16:33:14 MDT 2004
Windows NT 4.0 Service Pack 6a + security patches upto April 2004
---------------------oOo----------------------
<<<<<Here is the experiment ... annotated>>>>>
****** Clear away previous attempts *************
[mark@turing mark]$ cd /home2/video
[mark@turing video]$ rm Seb* test.avi
rm: cannot lstat `Seb*': No such file or directory
rm: cannot lstat `test.avi': No such file or directory
******* Disk drives and partitions used in test ***********
******* /mnt/win_d2 /home/mark /home2/video were used *****
******* win_d2 and video are on the same drive *******
[mark@turing video]$ df
Filesystem Size Used Avail Use% Mounted on
/dev/ide/host0/bus0/target0/lun0/part5
4.9G 2.7G 2.0G 58% /
/dev/ide/host0/bus1/target0/lun0/part5
5.1G 3.4G 1.5G 70% /home
/dev/ide/host0/bus0/target1/lun0/part5
5.8G 2.7G 3.2G 46% /home2
/dev/ide/host0/bus0/target0/lun0/part1
4.9G 3.2G 1.8G 64% /mnt/win_c
/dev/ide/host0/bus0/target1/lun0/part1
9.8G 4.7G 5.2G 48% /mnt/win_c2
/dev/ide/host0/bus1/target0/lun0/part1
479M 457M 23M 96% /mnt/win_c3
/dev/ide/host0/bus0/target0/lun0/part6
5.9G 4.5M 5.9G 1% /mnt/win_d
/dev/ide/host0/bus0/target1/lun0/part2
20G 15G 5.5G 73% /mnt/win_d2
/dev/ide/host0/bus0/target0/lun0/part7
5.9G 4.6G 1.3G 79% /mnt/win_e
/dev/ide/host0/bus0/target0/lun0/part8
17G 5.5G 11G 34% /mnt/win_f
/dev/ide/host0/bus0/target0/lun0/part2
20G 7.5G 13G 39% /mnt/win_g
******* Copy 1GB file from NTFS partition to ext3 partition *********
******* Do it twice to show there is a random character to the problem *****
******* Note that source and destination are on same drive but different
partitions ****
[mark@turing video]$ cp /mnt/win_d2/Seb* .
[mark@turing video]$ cp /mnt/win_d2/Seb* ./test.avi
[mark@turing video]$ ls -l /mnt/win_d2/Seb* ./Seb* ./test.avi
-r-xr-xr-x 2 root root 1075496620 Oct 3 22:41 /mnt/win_d2/Seb at Kiotari
2001.avi*
-r-xr-xr-x 1 mark mark 1075496620 Oct 6 21:02 ./Seb at Kiotari 2001.avi*
-r-xr-xr-x 1 mark mark 1075496620 Oct 6 21:05 ./test.avi*
****** Now compare files with the original ******
[mark@turing video]$ cmp /mnt/win_d2/Seb* ./Seb*
/mnt/win_d2/Seb at Kiotari 2001.avi ./Seb at Kiotari 2001.avi differ: byte
35194881, line 209008
[mark@turing video]$ cmp /mnt/win_d2/Seb* ./test.avi
/mnt/win_d2/Seb at Kiotari 2001.avi ./test.avi differ: byte 10455041, line
33722
[mark@turing video]$ cmp Seb* test.avi
Seb at Kiotari 2001.avi test.avi differ: byte 10455041, line 33722
******* Why does cmp find differences? *******
******* How does cp read and write files? ****
******* Is it different from the way cmp reads files? ***
[mark@turing video]$
****** Let us try copying between two drives *****
[mark@turing video]$ cp /mnt/win_d2/Seb* /home/mark
[mark@turing video]$ cmp /mnt/win_d2/Seb* ~/Seb*
/mnt/win_d2/Seb at Kiotari 2001.avi /home/mark/Seb at Kiotari 2001.avi differ:
byte 65937409, line 416721
***** Do copy between ext3 source and ext3 destination to show not electronic
problem ****
[mark@turing video]$
[mark@turing video]$ rm ./test.avi
rm: remove write-protected regular file `./test.avi'? y
[mark@turing video]$ cp ./Seb* ./test.avi
[mark@turing video]$ cmp Seb* test.avi
[mark@turing video]$
***** I conclude that file copy transactions from NTFS partition to Ext3 *****
***** are being corrupted. When I set the option to list the differences ****
***** I get a feeling that read and write buffers are not being manged
correctly in the presence of interrupt driven/ realtime processing ****
***** Problem may therefore not lie within the scope of the NTFS driver *****
<<<<<Experiment ends>>>>>
-----------------oOo---------------------
<<< dmesg output >>>
[mark@turing mark]$ dmesg
Linux version 2.6.3-16mdk (qa...@up...) (gcc version 3.3.2
(Mandrake Linux 10.0 3.3.2-6mdk)) #1 Fri Aug 13 16:33:14 MDT 2004
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000000bff0000 (usable)
BIOS-e820: 000000000bff0000 - 000000000bff3000 (ACPI NVS)
BIOS-e820: 000000000bff3000 - 000000000c000000 (ACPI data)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
191MB LOWMEM available.
On node 0 totalpages: 49136
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 45040 pages, LIFO batch:10
HighMem zone: 0 pages, LIFO batch:1
DMI 2.2 present.
ACPI disabled because your bios is from 99 and too old
You can enable it with acpi=force
Built 1 zonelists
Kernel command line: root=/dev/hda5 devfs=mount acpi=ht resume=/dev/hdc6
splash=silent vga=791
bootsplash: silent mode.
No local APIC present or hardware disabled
Initializing CPU#0
PID hash table entries: 1024 (order 10: 8192 bytes)
Detected 350.832 MHz processor.
Using tsc for high-res timesource
Console: colour dummy device 80x25
Memory: 190544k/196544k available (1817k kernel code, 5384k reserved, 848k
data, 272k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay loop... 690.17 BogoMIPS
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
checking if image is initramfs...it isn't (no cpio magic); looks like an
initrd
Freeing initrd memory: 136k freed
CPU: After generic identify, caps: 008021bf 808029bf 00000000 00000000
CPU: After vendor identify, caps: 008021bf 808029bf 00000000 00000000
CPU: L1 I Cache: 32K (32 bytes/line), D cache 32K (32 bytes/line)
CPU: After all inits, caps: 008021bf 808029bf 00000000 00000002
CPU: AMD-K6(tm) 3D processor stepping 0c
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
NET: Registered protocol family 16
EISA bus registered
PCI: PCI BIOS revision 2.10 entry at 0xfb470, last bus=1
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040211
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
PnPBIOS: Disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Using IRQ router VIA [1106/0586] at 0000:00:07.0
vesafb: framebuffer at 0xe3000000, mapped to 0xcc800000, size 4096k
vesafb: mode is 1024x768x16, linelength=2048, pages=0
vesafb: protected mode interface info at c000:0310
vesafb: scrolling: redraw
vesafb: directcolor: size=0:5:6:5, shift=0:11:5:0
fb0: VESA VGA frame buffer device
apm: BIOS version 1.2 Flags 0x07 (Driver version 1.16ac)
ikconfig 0.7 with /proc/config*
VFS: Disk quotas dquot_6.5.1
devfs: 2004-01-31 Richard Gooch (rg...@at...)
devfs: boot_options: 0x1
Initializing Cryptographic API
Activating ISA DMA hang workarounds.
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
bootsplash 3.1.4-2004/02/19: looking for picture... no good signature found.
Console: switching to colour frame buffer device 128x48
pty: 1024 Unix98 ptys configured
Serial: 8250/16550 driver $Revision: 1.90 $ 20 ports, IRQ sharing enabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 32000K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: IDE controller at PCI slot 0000:00:07.1
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: VIA vt82c586b (rev 47) IDE UDMA33 controller on pci0000:00:07.1
ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:DMA, hdb:DMA
ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:DMA, hdd:DMA
hda: IC35L060AVER07-0, ATA DISK drive
hdb: ST3120026A, ATA DISK drive
Using anticipatory io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdc: QUANTUM FIREBALL EX6.4A, ATA DISK drive
hdd: LG CD-RW CED-8120B, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 120103200 sectors (61492 MB) w/1916KiB Cache, CHS=65535/16/63, UDMA(33)
/dev/ide/host0/bus0/target0/lun0: p1 p2 p3 < p5 p6 p7 p8 >
hdb: max request size: 1024KiB
hdb: 234441648 sectors (120034 MB) w/8192KiB Cache, CHS=16383/255/63, UDMA(33)
/dev/ide/host0/bus0/target1/lun0: p1 p2 p3 < p5 >
hdc: max request size: 128KiB
hdc: 12594960 sectors (6448 MB) w/418KiB Cache, CHS=13328/15/63, UDMA(33)
/dev/ide/host0/bus1/target0/lun0: p1 p2 < p5 p6 >
mice: PS/2 mouse device common for all mice
serio: i8042 AUX port at 0x60,0x64 irq 12
input: ImPS/2 Generic Wheel Mouse on isa0060/serio1
serio: i8042 KBD port at 0x60,0x64 irq 1
input: AT Translated Set 2 keyboard on isa0060/serio0
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
EISA: Probing bus 0 at eisa0
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 16384 bind 32768)
NET: Registered protocol family 1
BIOS EDD facility v0.13 2004-Mar-09, 3 devices found
Please report your BIOS at http://linux.dell.com/edd/results.html
Resume Machine: resuming from /dev/hdc6
Resuming from device hdc6
Resume Machine: This is normal swap space
PM: Reading pmdisk image.
PM: Resume from disk failed.
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem).
Mounted devfs on /dev
SCSI subsystem initialized
PCI: Found IRQ 9 for device 0000:00:0b.0
scsi0 : AdvanSys SCSI 3.3GJ: PCI Ultra: IO 0xE400-0xE40F, IRQ 0x9
Mounted devfs on /dev
Freeing unused kernel memory: 272k freed
Real Time Clock Driver v1.12
drivers/usb/core/usb.c: registered new driver usbfs
drivers/usb/core/usb.c: registered new driver hub
drivers/usb/host/uhci-hcd.c: USB Universal Host Controller Interface driver
v2.1
uhci_hcd 0000:00:07.2: UHCI Host Controller
uhci_hcd 0000:00:07.2: irq 10, io base 0000d400
uhci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 1
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
usb 1-2: new full speed USB device using address 2
hub 1-2:1.0: USB hub found
hub 1-2:1.0: 4 ports detected
Adding 408200k swap on /dev/hdc6. Priority:-1 extents:1
imm: Version 2.05 (for Linux 2.4.0)
parport0: PC-style at 0x378 [PCSPP,TRISTATE,EPP]
parport0: cpp_daisy: aa5500ff(08)
parport0: assign_addrs: aa5500ff(08)
parport0: cpp_daisy: aa5500ff(08)
parport0: assign_addrs: aa5500ff(08)
ppa: Version 2.07 (for Linux 2.4.x)
Initializing USB Mass Storage driver...
drivers/usb/core/usb.c: registered new driver usb-storage
USB Mass Storage support registered.
Linux agpgart interface v0.100 (c) Dave Jones
agpgart: Detected VIA Apollo MVP3 chipset
agpgart: Maximum main memory to use for agp memory: 149M
agpgart: AGP aperture is 16M @ 0xe2000000
kjournald starting. Commit interval 5 seconds
EXT3 FS on hdc5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on hdb5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Supermount version 2.0.4 for kernel 2.6
NTFS driver 2.1.6 [Flags: R/O MODULE].
NTFS volume version 1.2.
NTFS volume version 1.2.
NTFS volume version 1.2.
NTFS volume version 1.2.
NTFS volume version 1.2.
NTFS volume version 1.2.
NTFS volume version 1.2.
8139too Fast Ethernet driver 0.9.27
PCI: Found IRQ 11 for device 0000:00:0a.0
IRQ routing conflict for 0000:00:0a.0, have irq 9, want irq 11
eth0: RealTek RTL8139 at 0xccd16000, 00:40:f4:44:71:98, IRQ 9
eth0: Identified 8139 chip type 'RTL-8139C'
inserting floppy driver for 2.6.3-16mdk
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
hdd: ATAPI 32X CD-ROM CD-R/RW drive, 8192kB Cache, DMA
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
drivers/usb/core/usb.c: registered new driver hiddev
drivers/usb/core/usb.c: registered new driver hid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
NET: Registered protocol family 17
eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
PCI: Found IRQ 5 for device 0000:00:09.0
lp0: using parport0 (polling).
lp0: console ready
mtrr: 0xe3000000,0x1000000 overlaps existing 0xe3000000,0x400000
eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
[mark@turing mark]$
-------------oOo---------------
<<<<fstab output>>>>>
[mark@turing mark]$ cat /etc/fstab
/dev/hda5 / ext2 defaults 1 1
none /dev/pts devpts mode=0620 0 0
/dev/hdc5 /home ext3 defaults 1 2
/dev/hdb5 /home2 ext3 defaults 1 2
/dev/hdd /mnt/cdrom auto
umask=0,user,iocharset=iso8859-15,codepage=850,noauto,ro,exec 0 0
none /mnt/floppy supermount
dev=/dev/fd0,fs=ext2:vfat,--,umask=0,iocharset=iso8859-15,sync,codepage=850 0
0
/dev/hda1 /mnt/win_c ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hdb1 /mnt/win_c2 ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hdc1 /mnt/win_c3 vfat umask=0,iocharset=iso8859-15,codepage=850 0 0
/dev/hda6 /mnt/win_d ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hdb2 /mnt/win_d2 ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hda7 /mnt/win_e ntfs umask=0,nls=iso8859-15,ro 0 0
#/dev/hdb3 /mnt/win_e2 ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hda8 /mnt/win_f ntfs umask=0,nls=iso8859-15,ro 0 0
/dev/hda2 /mnt/win_g ntfs umask=0,nls=iso8859-15,ro 0 0
none /proc proc defaults 0 0
/dev/hdc6 swap swap defaults 0 0
none /mnt/removable supermount
dev=/dev/scsi/host1/bus0/target0/lun0/part1,fs=ext2:vfat,--,umask=0,iocharset=iso8859-15,kudzu,codepage=850
0 0
[mark@turing mark]$
|
|
From: Anton A. <ai...@ca...> - 2004-10-07 13:02:19
|
Hi, On Thu, 2004-10-07 at 00:00, mark wrote: > The problem I am experiencing was detected when I was making a Super Video CD > from a 5 minute video produced using Premiere under Windows NT4 SP6a. The > problem manifested itself as visual artefacts. I have traced the problem > back to the copying of a large *.avi file from an NTFS directory to an Ext3 > folder. The size of the file is the same, but the content differs. > Repeating the test generates errors in different parts of the file. If I > were to take a look at the detailed "cmp" output I would suspect that buffer > management is not properly managed across the interrupt/realtime part and the > user priority part... It looks as though a block has been added or removed (I > cannot fathom which). No, the blocks just get corrupted. > I don't know how to proceed analysing the problem any further. You shouldn't need to. > Linux version 2.6.3-16mdk (qa...@up...) (gcc version 3.3.2 > (Mandrake Linux 10.0 3.3.2-6mdk)) #1 Fri Aug 13 16:33:14 MDT 2004 > > Windows NT 4.0 Service Pack 6a + security patches upto April 2004 [snip] > ***** I conclude that file copy transactions from NTFS partition to Ext3 ***** > ***** are being corrupted. When I set the option to list the differences **** > ***** I get a feeling that read and write buffers are not being manged > correctly in the presence of interrupt driven/ realtime processing **** > > ***** Problem may therefore not lie within the scope of the NTFS driver ***** Correct. This has nothing to do with ntfs at all and it is a known early 2.6 kernel problem and has been fixed ages ago (well a month ago at least I think) but Mandrake seem to be using a very old 2.6 kernel which appears to still have the problem. Note that you are not actually using Mandrake's latest kernel so you might want to upgrade to that and try again. If the problem persists file a bug report with Mandrake and tell them to upgrade their kernel or find the bugfix and back port it. It is possible you are hitting a bug different to the one that was fixed that simply causes the file to be corrupted somehow. One possibility is if the file in Windows was compressed in which case you need at least NTFS driver version 2.1.12 to ensure that it is read correctly. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ |
