Re: [Pfspd-users@sf.net] performance problems

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Rene, Bram,

Write performance seems to be dependent on access mode.
I think "rb+" implies read-modify-write of chunks of the file, hence=20
slower speed.

The performance problem is due to the brain-dead win32 API, where we have
to open a file in non-overlapped mode for allocation, close it, then open=20
in overlapped mode for performance.
For simplicity, we did the same for Linux (open, allocate, close, open),=20
but this meant the file was reopened in
update mode (otherwise it would be truncated to 0 bytes).

I fixed the issue with conditional compilation (#ifdef FIO=5FWIN32=5FFILE)
For Linux, I put the file allocation to the regular spot (after file=20
open), so the access
mode can be "wb" (write) or "rb+" (update), as specified by the rewrite=20
flag.
For Win32, allocation is done first, followed by a re-open in update mode.

Please verify if this solves your performance problems.
Regards,
Harold
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Ing. H.A.W. Schmeitz
Video Processing Systems, Philips Research Laboratories

High Tech Campus 36, 5656 AE Eindhoven, The Netherlands
Tel. +31 (0)40 27 46489
har...@ph...

Rene van der Vleuten=20
Sent by:
pfs...@li...
09-06-2006 12:59

To
Bram Riemens/EHV/RESEARCH/PHILIPS@PHILIPS
cc
pfs...@li...
Subject
Re: [Pfs...@sf...] performance problems
Classification

Hi!=20

Without being an expert (and without having looked at the code), I would=20
expect you could use append mode while writing data and then open in=20
update mode when the file is closed in order to correct the header (i.e. I =

would not update the header after each frame has been written, assuming=20
that is what is currently happening).=20

Regards,=20

Ren=E9.=20

--------------------------------------------------------------------------
Dr. Rene J. van der Vleuten
Philips Research Laboratories
High Tech Campus 36
5656 AE  Eindhoven
The Netherlands

Phone: +31 40 2742941
Fax:   +31 40 2742630
mailto:Ren...@ph...=20

Bram Riemens=20
Sent by:=20
pfs...@li...=20
08-06-2006 17:06=20

To
pfs...@li...=20
cc
Gerard Bloemen/EHV/RESEARCH/PHILIPS@PHILIPS=20
Subject
Re: [Pfs...@sf...] performance problems=20
Classification

Rene, others,=20

After obtaining a test case from Rene, I did some investigations on the=20
current development=20
repository of cpfspd and pts.=20
Test conditions:=20
- linux executable, running on dimholt=20
- output to scratch disk on muscules file server=20
- command "pts translate corvette=5F8mbps.m2v corvette=5F8mbps.yuv"=20
- result is file of 3.6Gbyte; 1198 frames 1920x1080=20

Currently (development version of cpfspd), file is opened in mode "wb" for =

allocation,=20
then it is closed. With pts translate on mpeg, the file size is not known=20
in advance,=20
so it is initially created with size 22528 bytes.=20
After that, it is reopened in "rb+" mode. Then images are written into the =

file.=20
Result of "time":=20
real    18m11.415s=20
user    1m10.260s=20
sys     1m2.590s=20

As a special test, I forced opening the file in "wb" mode the 2nd time;=20
result:=20
real    2m23.710s=20
user    1m5.710s=20
sys     0m20.850s=20

Next test; force mode "wb+" is also very slow.=20

I tested opening in append modes (followed by a reset of the file=20
pointer), but that fails to give correct output.=20

So, just changing the fopen mode results in a performance penalty of a=20
factor 7.5...=20
Interesting observation...=20

Harold, any suggestion how to keep the file open, and still maintain a=20
simple interface=20
for both *nix and win32/cygwin?=20

Regards, Bram.=20

PS. Gerard, maybe this is related to your observations?=20
       What version did you use?=20
       You can subscribe this mailing list at=20
               https://lists.sourceforge.net/lists/listinfo/pfspd-users

--
A.K. (Bram) Riemens
Principal Scientist, DSP group, Philips Research
Office: WO-p-94, Postbox WO02
High Tech Campus 36 (WO), 5656 AE Eindhoven, The Netherlands
Tel: +31 40 27 43833, Fax: +31 40 27 44675
E-mail: bra...@ph...=20

Rene van der Vleuten=20
Sent by:=20
pfs...@li...=20
08-06-2006 13:04=20

To
pfs...@li...=20
cc

Subject
[Pfs...@sf...] performance problems=20
Classification

Dear Bram, Harold,=20

I am suffering from huge performance problems with "pts translate file.m2v =

file.yuv"; processing takes more than 10x longer than with the latest=20
released pts!! I suspect these are caused by your latest updates to cpfspd =

(since I did not see any obviously related changes in pts).=20

Could you please look into this?=20

Thanks!=20

Ren=E9=20

--------------------------------------------------------------------------
Dr. Rene J. van der Vleuten
Philips Research Laboratories
High Tech Campus 36
5656 AE  Eindhoven
The Netherlands

Phone: +31 40 2742941
Fax:   +31 40 2742630
mailto:Ren...@ph...=20
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Pfspd-users mailing list
Pfs...@li...
https://lists.sourceforge.net/lists/listinfo/pfspd-users
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Pfspd-users mailing list
Pfs...@li...
https://lists.sourceforge.net/lists/listinfo/pfspd-users
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Pfspd-users mailing list
Pfs...@li...
https://lists.sourceforge.net/lists/listinfo/pfspd-users