File 'cpfspd.tgz' has been built with the following command
tar/gtar -zcf cpfspd.tgz cpfspd
To unpack this file, execute
tar/gtar -zxf cpfspd.tgz
This will create a directory tree 'cpfspd'.
------------- Start of cpfspd/Readme.txt file -------------
Copyright (c) KONINKLIJKE PHILIPS ELECTRONICS N.V. 1997-2015
Copyright (c) NXP Semiconductors Netherlands B.V. 2006-2010
All rights reserved.
For licensing and warranty information, see the file COPYING.LGPLv21.txt
Philips Research Laboratories
Eindhoven, The Netherlands
CVS id : $Id: Readme.txt,v 2.196 2015/10/20 12:13:01 vleutenr Exp $
Library : cpfspd
Purpose : Interface to the PFSPD file format
Download : http://sourceforge.net/projects/pfspd
Website : http://pfspd.sourceforge.net
Maillist : pfspd-users@lists.sourceforge.net subscribe via
http://lists.sourceforge.net/mailman/listinfo/pfspd-users
Authors : Robert Jan Schutten
Bram Riemens
Release : 1.12.4
Date : 20-OCT-2015
License
-------
The CPFSPD library is licensed under the terms of version 2.1 of the GNU
Lesser General Public License as published by the Free Software Foundation.
The PV and PP applications are licensed under the terms of version 2 of the
GNU General Public License as published by the Free Software Foundation.
The MTK package contains some support utilities which is also available
under the license term of version 2 of the GNU General Public License.
The license text can be found in the top level directories of the
respective packages.
Purpose
-------
The reason to write this library is twofold:
- Provide an access to pfspd files in ansi-C (so with known source
code), in order to be able to use the pfspd file format on every
platform that supports an ansi-C compiler.
This is achieved with the cpfspd_low.c file.
- Provide an easy to understand interface (at least for humans) to
access pfspd files, without the need to have deep knowledge about
the header structure of the pfspd file format.
However, it should still be possible to modify the header structure
directly for functionality not provided in this library.
This is achieved with the other cpfspd_???.c files.
Later, additional requirements developed:
- Large file support (that is: files with size larger than 2 Gbyte)
on all platforms/file systems that support this. On other platforms,
the software does still work within the limits of the platform.
This is achieved by conditional compilation of cpfspd_fio.c.
It is completely transparant for the application programmer.
- Efficient file access on the MSwindows platform. With the increasing
performance of windows based PC's, more applications are developed
with real-time streaming requirements. This is also achieved
with conditional compilation in cpfspd_fio.c. With proper compilation
settings, windows specific file access routines are used to perform
unbuffered asynchronous disk operations. Buffering is done by the
cpfspd_fio module. As a result, there is only a single copy operation
to move the image data to the application buffer.
It is completely transparant for the application programmer.
General approach in the design of cpfspd:
+ Access to standard video files should be really easy.
+ More unusual video formats shall still be supported,
but more specific knowledge and programming effort will be required.
Short explanation of features:
(copied from cpfspd.h)
PFSPD = Philips File Standard for Pictorial Data
The routines in this file provide an interface to
pfspd video files.
The routines in this file provide the following
functionality:
- Reads/writes:
+ progressive video files
+ interlaced video files
+ access per field and/or frame
- Reads/writes:
+ luminance only
+ 4:4:4 YUV (planar UV)
+ 4:2:2 YUV (multiplexed or planar UV)
+ 4:2:0 YUV (multiplexed or planar UV)
+ 4:4:4 RGB
+ 4:4:4 XYZ
+ streaming files ("S" files, e.g. cvbs encoded files)
- File formats supported:
+ 8, 10, 12, 14 & 16 bit per pixel files
(pfspd types: B*8, B*10, B*12, B*14, I*2).
- Memory formats supported:
+ 8 bits (unsigned char, with 8 bits data)
+ 16 bits (unsigned short, with 8, 10, 12, 14
or 16 bits data)
- Extra components with application defined name,
independent file format and subsample factors.
- Easy header definition and modification for
Standard Definition and several High Definition
formats (including all formats defined by ATSC)
Usage
-----
Read the file cpfspd.h for a detailed description of all functionality
that is included in this library.
Example usage to read a pfspd file:
int main(int argc, char *argv[])
{
char *filename = "example.yuv";
pT_header header;
pT_status status;
unsigned char y_fld[720*288];
unsigned char uv_fld[720*288];
/* read pfspd file header */
status = p_read_header(filename, &header);
/* exit on error */
p_fatal_error(status, stderr);
/* read first field (assuming file is interlaced & contains color) */
status = p_read_field(filename,
&header,
1, /* first frame */
1, /* first field */
y_fld, /* Y buffer */
uv_fld, /* U/V buffer */
P_READ_ALL, /* read Y and U/V */
720, /* width */
288, /* height */
720); /* stride */
/* exit on error */
p_fatal_error_fileio(status, filename, stderr);
}
Example usage to write a pfspd file:
int main(int argc, char *argv[])
{
char *filename = "example.yuv";
pT_header header;
pT_status status;
unsigned char y_fld[720*288];
unsigned char uv_fld[720*288];
/* fill y_fld & uv_fld with useful data */
/* create new file header; 4:2:2 color & 50 Hz */
/* N.B. number of frames in file is zero */
status = p_create_header(&header, P_COLOR_422, P_50HZ);
/* exit on error */
p_fatal_error(status, stderr);
/* set number of frames in file to 1 */
status = p_mod_num_frames(&header, 1);
/* exit on error */
p_fatal_error (status, stderr);
/* room to modify header to double progressive or double image rate */
/* e.g. use: p_mod_to_progressive(&header); */
/* e.g. use: p_mod_to_dbl_image_rate(&header); */
/* create file on disk & write header */
status = p_write_header(filename, &header);
/* exit on error */
p_fatal_error_fileio(status, filename, stderr);
/* write first field */
status = p_write_field(filename,
&header,
1, /* first frame */
1, /* first field */
y_fld, /* Y buffer */
uv_fld, /* U/V buffer */
720, /* width */
288, /* height */
720); /* stride */
/* exit on error */
p_fatal_error_fileio(status, filename, stderr);
}
Supported file formats
----------------------
Obviously, since this is the "core" pfspd access library this supports
all identified formats:
Color formats:
Only Y data (luminance only).
YUV 4:2:2 MULTIPLEXED, (as defined by ITU-R BT. 601-4) with multiplexed UV.
YUV 4:2:0 MULTIPLEXED, (as defined by mpeg) with multiplexed UV.
YUV 4:2:2 PLANAR, with planar storage of U and V.
YUV 4:2:0 PLANAR, with planar storage of U and V.
YUV 4:4:4 PLANAR, planar storage of U and V.
RGB data, by definition this is 4:4:4 planar.
XYZ data, by definition this is 4:4:4 planar.
Data formats:
8 Bit binary offset (as defined by ITU-R BT. 601-4).
10,12,14 and 16 bit binary offset.
Extra components:
An extra component is defined by:
- name
- data format (independent of the standard components)
- subsample factor
Compilation
-----------
In almost any situation, compilation is not required. The standard
distribution contains pre-compiled libraries for many systems.
It is always preferred to link a pre-compiled library with your
application. For those who want to execute on exotic systems
or experiment their own modifications, follow these guidelines:
- On make based systems (e.g. unix & cygwin):
Type "gmake" (or "make" if that is the gnu-make utility, e.g. on
Linux and Cygwin) in the src directory.
- Native windows:
The make system uses standard unix commands like "sed" and "cksum"
to generate magic numbers, readme text etc. When cygwin is installed,
these unix commands are available. Use "make COMP=msvc" to build cpfspd
with the MS Visual C commandline tools. Solution files are provided for
Release and Debug builds for multiple VisualStudio versions.
- Other systems:
Copy all source and header files to your project.
Make sure to pass the version and magic number information
to cpfspd_mag.c using C preprocessor defines.
If large file support is required, then make sure to
pass the required conditional compilation settings to
module cpfspd_fio.c.
File open/close and file length
-------------------------------
In order to optimize the file access, the cpfspd library opens
and closes files as follows (default behavior):
- A file is opened at its first access.
- A file is closed at program termination.
- At most 10 files are opened simultaneously;
if a next file needs to be opened, the least recently used file
is closed.
This behavior can be controlled by the application when
it opens and closes the files itself.
In the pfspd file format, the number of images in the file are
registered in the header. Applications can deal with the file
length in two ways:
1) Specify the length at the beginning, when writing the header.
2) Specify zero length in the header, and start writing.
When specifying the length before the header is written,
the header is correct when it is written to file. In this case,
cpfspd determines the complete file size and allocates the required
disk space immediately. This assures that all data can actually be
written to disk. Since the whole file is allocated at once, this
also prevents file fragmentation on NTFS file systems.
When the application writes more data into the file than
initially specified, this is registered by cpfspd. At the
time that the file is closed, the file length is corrected
(i.e. the header is rewritten at the beginning of the file).
In case standard I/O via pipes is required by the application,
it needs to assure that the initial header has the correct length.
Pipes do not allow random access, only sequential access.
File allocation on harddisk
---------------------------
When executing p_write_header(), not only the header is written
to disk, but also all frames (as indicated in the header) are
allocated. This is done by opening the file, seeking to the desired
file size position and setting the end-of-file mark. This is convenient,
since problems with file size (e.g. due to quota limitations)
appear at program start, not after hours of computations...
On unix systems, sparse files are supported by default: only data that
is actually written is physically allocated on disk. Hence, this
"file allocation" executes very fast since only the end-of-file mark
is set, but no actual data is written (apart from the header itself).
Windows systems with a local NTFS file system, however, behave entirely
different. By default, 'holes' in a file are automatically filled with
zero data. Hence, the standard strategy of file allocation results in
the complete file being filled with zeros. This may take a considerable
amount of time. We experimented with sparse files in NTFS, but that
leads to considerable file fragmentation (a 10G file was normally
stored in a few dozen of fragments; after setting the file sparse
this increased to 3000 a 4000 fragments). This is caused by the NTFS
allocation policy to locate the smallest free area for an allocation
request. In a sparse file, each extension of the file is a new
allocation request. An alternative solution was implemented, using
a special windows API function call SetFileValidData(). This just
informs the file system that all data up to the end can be considered
written (hence, skipping the need for zero-filling). This may involve
security issues, and therefore the function requires an elevated
priveledge that is only granted for accounts with administrator rights.
The security risk is that this function may expose data that already
resides on the disk (originating from deleted files). This risk can be
alleviated by assuring that the application actually writes all frames
announced in the header. Finally, we found that the function fails
in a cygwin version 1.5.25-15 environment; this is resolved in version
1.7.1.
In summary, for fast file allocation on NTFS file systems:
- execute with administrator rights
- when using a cygwin exec, install cygwin version 1.7.x
- write all frames (and all components) announced in the header
In case the special priveledge is not granted, the cpfspd library
silently falls back to the original behavior of zero-filling the
file.
Multi-threading applications
----------------------------
The functions in cpfspd are not reentrent. The library keeps
track of open files by maintaining some global data internally.
Simultaneous attempts to access this administration may
corrupt the data. However, only reading this data is harmless.
The data is modified when a file is opened or closed.
In a multi threading application one of the following two
approaches must be followed:
1) First open all files in a single thread, then start threads
to process the data from the opened files. More precise:
- all open and close operations shall be performed in a single thread
- a single file shall be accessed only within a single thread
- no more than 10 files shall be accessed
2) Perform all accesses to a file in a single thread and
protect all file open and close operations as critical sections.
More precise:
- all open and close operations shall be sequentialized by putting
them in a critical section (e.g. guarded by a single semaphore)
- a single file shall be accessed only within a single thread
- no more than 10 files shall be accessed simultaneously
Ease of use versus functionality
--------------------------------
With the cpfspd library, there are three levels of complexity:
1. Straightforward image access of standard files
2. Use of additional components
3. Arbitrary components without checking
The more advanced use level one chooses, the more programming
and knowledge of the pfspd standard is required; at the same time
the protection and checking of the cpfspd library is reduced.
At level 1, all standard functions can be used:
- p_<read|write>_<field|frame>[_planar][_16]
At level 2, the component access functions are available in addition
to the "standard" function of level 1. So these functions can be used:
- p_<read|write>_<field|frame>[_planar][_16]
- p_<read|write>_comp[_planar][_16]
At level 3, it is possible to define pfspd files with arbitrary
components. As a result, the standard functions of level 1 can NOT
be used! The application programmer is responsible for a correct
header definition. In order to use these features, the normal
header checking of the cpfspd library must be disabled.
This is achieved by setting the flag "disable_hdr_checks" in
the header structure. Further, only the component access functions
can be used to access image data:
- p_<read|write>_comp[_planar][_16]
This must be done before writing the header to file.
When reading a header of an arbitrary file, the function
p_read_header will return error code P_ILLEGAL_COLOR_FORMAT.
This must be ignored. After reading the header, the "disable_hdr_checks"
flag can be set.
Files created in this way cannot be handled by most tools!
Note: <x|y> denotes a choice between x and y; [z] is optional
Harddisk performance
--------------------
The cpfspd package has several features for disk access. Some
of these depend on the target platform. The table below lists
which features are available on the platforms.
Large file support allows access to files > 2Gbyte. Actual
availability of this feature also depends on the version of
the OS and type of file system. It does not exist on windows95/98/ME,
nor on the FAT16/32 file systems. Unix-like computers need to
be properly configured.
Asynchronous file I/O takes advantage of DMA between harddisk and
memory. During the DMA of one transfer, the data of an other transfer
is processed. Hence, a ping-pong buffering strategy is applied.
This only effects performance of local file systems; on network disks,
the network is the determining performance factor. The asynchronous
transfer is also unbuffered, i.e. the operating system does not perform
buffering, but data is transferred directly from harddisk to a cpfspd buffer.
From this buffer, only a single copy operation is carried out to put the
data in the application buffer.
Fast memcpy reduces the CPU load during the read/write of an image.
This is particularly important for multi-threading application
that need CPU capacity in one thread while the file access is perfomed
in an other thread.
Feature HP-UX IRIX64 Linux Linux64 SunOS cygwin win32 win64
Large file x x x x x x x x
Asynchronous x x x
Fast memcpy x x
For the asynchronous unbuffered I/O, cpfspd maintains a ping-pong buffer.
The default size of this buffer is 256 kbyte. The optimal size however
depends on the image size of an application. In general, the trade-off is
as follows:
- the larger the buffer, the more efficient the I/O (up to some limit)
- the smaller the buffer, the fewer unused data is read from disk
for random image access
So, for streaming applications, a large buffer size is generally an advantage.
Also for large image sizes, larger buffer sizes are advantageous.
Only in applications where harddisk bandwidth is critical, tuning
of the buffer size is advised. We suggest the following rule-of-thumb:
-> select the lowest power-of-2 that is larger than the largest component
E.g. for a real-time harddisk recorder bandwidth is critical,
when this is designed for HD signals, one Y component takes
approx. 2 Mbytes (1920x1080, 8 bits). In this case, a buffer size of
4096 would be appropriate. The buffer size is set at initialization using:
p_set_file_buf_size(const int size_kb)
Needless to say that the buffer size is only of interest on platforms
that support asynchronous I/O.
Wish list
---------
- Change function p_mod_defined_image_size to support all values of pT_freq.
- Add more test code for the cce (convenience) functions.
Done in release 1.12.4 (dd. October 20, 2015)
---------------------------------------------
Highlights
Fixed bug in conversion of small (subnormal) floating point numbers.
Added solutions for VisualStudio Release and Debug library builds.
Bug fixes
- p_cce_float_to_f16: Fixed bug in conversion of subnormal numbers
- p_write_aux_data: Fixed bug that prevented updating of aux data
Functionality
- None
Miscellaneous
- p_cce_check_float_conversion: All 16flt numbers are now checked
- Added solutions for VisualStudio Release and Debug library builds for multiple
VisualStudio versions (2008, 2010, 2012, and 2013)
Done in release 1.12.3 (dd. July 12, 2012)
---------------------------------------------
Highlights
Windows file I/O performance bug fixed.
Convenience functions for float <> f16 conversion made available
to end user.
Bug fixes
- Windows file I/O performance bug fixed. In some cases, reading a image
near the end of the file was very time-consuming (though correct data
was returned). Detailed explanation: jump to an image in the last buffer
and start reading there. This caused a ReadFile of the next buffer, which
happens to be located _after_ the end of the file. This "read ahead"
operation returned an eof flag, which was (erroneously) ignored. Hence,
every successive fread() operation caused this ReadFile attempt on the
file system, resulting in a huge overhead. In cpfspd, every video line
is read using a fread(). This is solved by taking the eof flag into
account when starting the "read ahead" ReadFile operation.
Functionality
- Convenience functions for HDR (float <> f16 conversions) made available
to end user
Miscellaneous
- None
Done in release 1.12.2 (dd. Januari 29, 2010)
---------------------------------------------
Highlights
Bugfix in use of float data for Linux64 platform.
Fast file allocation now also working for windows on local NTFS
file system.
Bug fixes
- gcc 64 uses 64-bit unsigned long, causing float conversion in p_cce
functions to fail. Unsigned longs replaced by uint32_t from stdint.h.
- In release 1.12.1, the Make.set file was 'forgotten', therefore
that release did not contain proper versioning information.
Functionality
- Fast file allocation on local windows NTFS file systems as described
a new section "File allocation on harddisk" in this readme file.
This works only for users with administrator rights. Further, on cygwin
at least version 1.7.x is required. On old cygwin versions or accounts
without administrator rights, cpfspd silently falls back to the old
behavior. As a side effect, this release requires at least OS version
Windows Server 2003 or Windows XP - older windows versions are not
supported anymore.
Miscellaneous
- Copyright date updated.
- Upgrade to mtk version 5.4.1 - compiler warnings in parse_options solved.
- Speed utility can also run Linux64 and win64 platforms.
- Speed utility has option -pfspd as synonym for the -yuv format for
improved clarity.
- Speed utility measures only active-duty time, i.e. time spend in opening
or closing files is not counted. This time is typically considered
application startup time, that is not in the performance-critical loop.
- Speed utility now has new option -prealloc. This controls whether the
complete file is allocated when writing the header (and before measuring
time). Previous behaviour was with -prealloc enabled. Note that the
consequence of this en the previous change is, that up till now, the
speed utility also counted the time of zero-filling the a file on NTFS.
- Filename for testing can be passed to the speed and buffer utilities.
- Added tests for the cce (convenience) functions with P_16_REAL_FILE format.
- The file allocation test is simplified; previously it tested two aspects:
1) initial file allocation when writing the header, and 2) writing a frame
at the end of the file without being allocated. Test 2) effectively tests
sparse files. This is not supported on NTFS, so on this file system, the
test still takes a lot of time. Test 2) is therefore removed.
- Internal simplification of cpfspd_fio.c: routine fio_fp_set_filepointer()
is substituted by direct calls of SetFilePointerEx().
- File size test in test.sh adapted to be robust against a group name
consisting of multiple words like "Domain Users".
Done in release 1.12.1 (dd. Novenber 19, 2009)
----------------------------------------------
Highlights
Added support for 64 bits code generation on windows
and Linux platforms as win64 and Linux64 targets.
Bug fixes
- None
Functionality
- None
Miscellaneous
- Copyright date updated.
- Upgrade of mtk and parse_options to support 64 bit code generation.
Now we support also the win64 and Linux64 platforms.
- Some code modifications in cpfspd_fio.c, test and speed programs
to handle 64 bit code generation.
Done in release 1.12.0 (dd. December 23, 2008)
----------------------------------------------
Highlights
New color format P_COLOR_XYZ, stored as integer or floating point.
New functions p_cce_read/write_float_xyz() for floating point XYZ files.
Several bugfixes; most notable: rounding in p_cce* functions.
Bug fixes
- Bug 1869155 resolved (warnings when compiling with gcc version 3.4.4-64 on
64 bit linux platform). Warnings:
cpfspd_low.c: In function `p_add_offset':
cpfspd_low.c:502: warning: left shift count >= width of type
cpfspd_low.c:511: warning: right shift count >= width of type
cpfspd_low.c: In function `p_position_pointer':
cpfspd_low.c:592: warning: left shift count >= width of type
cpfspd_low.c:653: warning: right shift count >= width of type
Cause: with the 64 bit versions of gcc, a long holds a 64 bit value.
The issue is resolved by forcing offset_hi/lo shifts as 32 bit shifts
(assuming a long will hold at least 32 bits).
Now, compilation on 64 bits gcc/linux platforms is supported, but not standard
practice for releases.
- Bug 1972938 resolved (data changed by p_cce_read/write_comp() functions).
Added rounding (+0.5) to the p_cce_read_comp() and p_cce_write_comp()
functions.
- Bug 1972967 resolved (p_cce_f16_to_float(0) == 1.0).
Used the correct exponent (0) to indicate an IEEE-754 zero (0.0).
- Bug 1992002 resolved (rounding errors for small float numbers)
The mantissa of denormalized numbers was not handled correctly.
- Bug in speed.c causing core dump by time stamps buffer overflow.
Functionality
- Color format P_COLOR_XYZ added. It can be used the same way as RGB.
XYZ data may also be saved as 16bit float components.
- RGB files may also be saved as 16bit float.
- p_cce_read/write_float_xyz() functions added.
These two functions are for convenience when working with XYZ data. All data
formats are accepted and converted from/to floating point application
buffers, which have a nominal range of [0..1].
Internally the p_cce_read/write_comp() functions are called with offset=0
and gain dependent on the data format (e.g. gain=255 for 8 bit data).
- The speed test utility has been extended with incremental bandwidth
measurement during writing and reading of pfspd files. Related option
"-numint" indicates the number of measurement intervals. Default numint is 1.
- Function p_rewrite_header() now accepts changes in width and height as long
as the total number of pixels remains consistent with the data in the file.
Miscellaneous
- Upgradt Make.mtk release 5.3.4 (support for MS Visual Studio 2008 / 9.0).
Done in release 1.11.0 (dd. April 28, 2008)
-------------------------------------------
Highlights
- Support for up to 128 components (was 15)
- Standard I/O on win32 supported
Bug fixes
- None
Functionality
- The cpfspd limit on number of compnents is changed from 15 to 128.
As a consequence, files with more than 15 components can be read and written.
When such file is created with a cpfspd application, it cannot be read
by the old tools using the pfspd-al interface layer.
- A new mem_data_fmt is added: P_AF_BIT_MEM for all image and component
read and write function on 16 bit buffers. This indicates that the
memory buffer uses the same word width as the file (AF: "As File").
- Use of standard I/O for the win32 target implemented.
Up till now, the use of stdio (and pipes) was not working on the win32
build. This was caused by the standard windows treatment of text files.
In the windows environent the standard input or standard output are
opened as text files and therefore subject to end-of-line substitution.
I.e. at input, \cr\lf is substituted by \cr and at output,
a \cr is substituted by \cr\lf. The cpfspd_low module contains
platform dependent settings to solve this.
Also the test script is adapted to incorporate testing of
stdio when selection the win32 target.
Thanks to Erik Kathman for providing the solution.
Miscellaneous
- Some explanation on buffer size added in the "Harddisk performance" section
of the readme file.
- In the speed directory, the 'time' info is removed from the test scripts
to improve readibility of the output.
- Consistency and cosmetics in cpfspd.h improved.
Done in release 1.10.2 (dd. Dec. 18, 2007)
------------------------------------------
Highlights
- Some bugfixes,
- When writing a file header, the whole file is now allocated
on disk in a very efficient way. Therefore, we recommend
application to open the file with the correct number of frames.
This results in 1) less file fragmentation on NTFS and
2) performance improvement on network disks and win32/cygwin
local disks.
- Performance improvement of >8 bit files.
Bug fixes
- Application segmentation fault on Linux.
Following calling sequence failed at the p_read_header() routine:
p_create_header(&out_header, P_COLOR_444_PL, P_50HZ);
p_write_header("test.yuv", &out_header);
p_close_file("test.yuv");
p_read_header("test.yuv", &out_header);
More general: the program may crash when accessing a closed file
in a different mode than it was previously used.
Underlaying cause: when accessing a file in a different mode,
it is first closed, then opened in the appropriate mode.
This test was incomplete: erreonously, a closed file is closed
again, effectively calling fclose(NULL). This is implementation
specific; some platforms ignore fclose(NULL).
- Bug #1506617: Growing network file with large chunk on win fails.
Situation: application executing on win32/cygwin, writing a file
on a network share; a frame is written a couple of Gbytes beyond
the higest written location in the file. The operation failes
with a timeout error.
A workaround is implemented, catching the timeout error and
reissuing the failed operation. The workaround is transparant
to the application.
- In the previous release, only writing files in big-endian
mode was supported. So, executing p_rewrite_header()
on a little-endian file was not supported. Unfortunately
cpfspd failed to signal this condition as an error.
Instead, the operation was carried out, silently modifying
the endian bit in the header to big-endian. This resulted
in incorrect data interpretation of files containing components
> 8 bits. This problem is resolved by supporting the creation
and modification of both big- and little-endian files.
Functionality
- The display aspect ratio setting can now have value P_AS_WH,
indicating that the intended aspect ratio of the diplay
is equal to the width/height ratio in pixels. This value
is an indication of square pixels as long as the width/height
ratio remains the same.
- Function p_create_ext_header() can now be called with
value P_UNKNOWN_ASPECT_RATIO in the ratio parameter.
This causes an automatic 16:9/4:3 selection.
See cpfspd.h for details on the selection algorithm.
Miscellaneous
- Module cpfspd_fio.c uses assembly code for cpu type checks
and speed optimization that is now also activated on cygwin
(formerly, this was only possible on win32).
- Files written on the windows (win32 or cygwin) platforms
contained unused space at the end due to the unbuffered
file I/O. As a consequence, the file length depended on
the platform. This is no longer the case. On win32/cygwin,
the file is truncated to the proper length.
- Due to best-fit allocation strategy of NTFS, pfspd files could become
severely fragmented, which interferes with playback performance.
This is fixed in p_write_hdr() by seeking to the required file size,
then writing a few bytes. NTFS now allocates a large extent,
so fragmentation is not a problem (unless the disk is almost full).
As a side benefit, performance on network disks is also improved.
Further, write performance on win32/cygwin takes full advantage
of asynchronous file I/O if the file is allocated in advance.
- Cpfspd will write 10-16 bit files according to the system endianness.
Prevously, all files were written with bit-endian data organization.
This allows optimization of the 16-bit read/write functions.
- The performance to access >8 bit files is improved.
In the cpfspd_low module, a copy operation to an intermediate buffer
is avoided if both:
+ memory format is equal to the file format
+ system endianness is equal to the file endianness
(for 8 bit files, this optimization was already available).
- Any output parameter of p_get_comp_2() may be NULL.
In case you need only a subset of the values (so you do not need
to allocate an output variable).
- New errors are introduced to catch header values that cannot be
represented in the pfspd file format: P_TOO_MANY_IMAGES,
P_ILLEGAL_INTERLACE, P_ILLEGAL_COMP_SIZE, P_ILLEGAL_LIN_SBSMPL,
P_ILLEGAL_PIX_SBSMPL, P_ILLEGAL_PHSHFT.
In order to support the new errors, all defines related to
header formatting are moved from cpfspd_low.c to cpfspd.h:
P_SNR_IMAGES, P_SNR_COMPON, P_SNR_FD_RECS, P_SNR_AUXDAT_RECS,
P_SBYTES_REC, P_SENDIAN_CODE, P_SAUX_HDR_RECS, P_LEN_GLOB_STR,
P_SIMA_FREQ, P_SLIN_FREQ, P_SPIX_FREQ, P_SACT_LINES, P_SACT_PIXEL,
P_SINTERLACE, P_SH_PP_SIZE, P_SV_PP_SIZE, P_LEN_GLOB_ATT,
P_NUM_GLOB_RECS, P_SLIN_IMAGE, P_SPIX_LINE, P_LEN_COMP_STR,
P_STEM_SBSMPL, P_SLIN_SBSMPL, P_SPIX_SBSMPL, P_LEN_COMP_ATT,
P_STEM_PHSHFT, P_SLIN_PHSHFT, P_SPIX_PHSHFT.
- Always use large file I/O when compiling with Visual C++.
It appears that many users compile the cpfspd library by simply including
the source code in their VC++ project, without defining FIO_WIN32_FILE in
the project properties. Hence, no large file support was compiled.
The FIO_WIN32_FILE define is now checked when __MSVC__ is defined.
- Speed test program uses HD image size (not SD anymore) as default.
- The speed test program puts the proper number of frames in the
header before writing the header to disk. This triggers the
new feature of fast disk allocation.
Done in release 1.10.1 (dd. Jan. 10, 2006)
------------------------------------------
Highlights
New release to recover from bugs in 1.10.0
Bug fixes
- Function p_write_header(), takes care that a newly written header
has room available for auxiliary headers (even when the header
supplied by the application does not request this). The calculation
of the amount of records was incorrect in cases where the record
length is not a power of 2. This is corrected. This bug was created
with the introduction of auxiliary data.
This bug caused the following error without appearent reason:
Error no: 228, description: Auxiliary header exceeds maximum size
- When reading from stdin using a pipe, all unused data is flushed
from the pipe before the program terminates to avoid a "broken pipe"
error. Previously, such error occured when an application wrote
more data into the pipe then it read from the pipe. This could easily
happen when writing extra components and reading only the standard
components.
Functionality
- None
Miscellaneous
- None
Done in release 1.10.0 (dd. Nov 14, 2005)
-----------------------------------------
Highlights
'Doxyfied'-code. First release for sourceforge.net
Added support for auxiliary data.
Bug fixes
- Uninitialized memory read occured in function p_mod_set_comp_2()
for a component name of 5 characters. This is fixed.
This was identified by purify checks. Actual damage has not been observed.
Functionality
- New set of functions for auxiliary data handling.
An arbitrary number of auxiliary data types can be specified.
The auxiliary data consists of a header with optional a data block per image.
The auxiliary headers are stored in the file description records (after the
regular file description). For compatibility with old pfspd applications,
the number of file description records now includes both file description
and auxiliary header records. The number of auxiliary header records is
stored separately in previously reserved space in the global data file
header record. The auxiliary data per image is stored in image header
records, which were defined in the pfspd file format, but not used by
the cpfspd interface.
For a more precise description, see the website (menu item "file format").
For this new functionality, the following changes were required:
+ In the pT_header structure, field nr_ih_recs is renamed to
nr_aux_data_recs.
In previous releases, cpfspd did not support "image header records".
When writing a file, the number of "image header records"
was always zero. When reading, they were skipped.
Now, this feature in the file format is used for multiple auxiliary
data items. Hence the field is renamed.
+ The pT_header structre is extended with a new field nr_aux_hdr_recs.
This field is stored in previously reserved space in the global data
file header record of the pfspd file format.
+ The pT_header structure is extended with a field aux_hdrs.
This holds the auxiliary header data.
- Added two new functions:
p_get_all_freqs(): gets image-, line-, and pixel frequencies
p_mod_all_freqs(): sets image-, line-, and pixel frequencies
Miscellaneous
- The package is released under the LGPL licence now.
- New mtk which uses single Make.set file for version number definition
of both lib and applications.
- Added Doxygen markup to cpfspd header
- New demonstration application added: demo_filter.
- New demonstration application added: demo_raw.
This also serves as a usefull utility to convert raw image
data to pfspd files and vice-versa.
- The number of TriMedia processor variants is increasing.
Compiled code is specific for a particular processor instruction
architecture. Since it is not convenient to keep all processor
variants precompiled in the release, it was decided to stop
supplying precompiled libraries for TriMedia cores as part of
cpfspd releases. Compilation for many TriMedia processors remains
supported by the makefile system. TriMedia users shall obtain a
copy of the cpfspd source code and compile themselves.
- Default TCS release at natlab updated to version 2.6pr1.
Done in release 1.9.4 (dd. March 31, 2005)
------------------------------------------
Highlights
Recommendation: use p_get_comp_2() i.s.o. p_get_comp().
Recommendation: use p_mod_set_comp_2() i.s.o. p_mod_set_comp().
Bug fixes
- Function p_cce_read_comp() behaved incorrect under the following conditions:
+ application buffer of type P_FLOAT or P_DOUBLE
+ gain setting > 1
+ integer type pfspd data
The sw used an integer division in stead of a float division.
So for 8 bits data and a gain of 255, the float values were rounded
to values 0.00 or 1.00 only.
This bug was introduced in release 1.9.3 as an unwanted side-effect
of the introduction of float data in pfspd files.
- A nasty subtle bug in the file access module cpfspd_fio is fixed.
The module behaved incorrectly under the following coditions:
+ platform is win32 or cygwin
+ file is opened in write-update mode (open call with mode "wb+")
+ when writing data, part of the file is left untouched (i.e. skipped)
+ the untouched part is within a single buffer of 256 kbytes
The untouched part was not filled in the buffer; at a later stage
the entire buffer was written to disk. As a result, the untouched
part of the file was overwritten with undefined data.
This bug is fixed and now the untouched part of the file is read into
the buffer to assure proper update behaviour.
We hit this bug when changing component properties with "pts header"
on a file with a large amount of description records.
- p_cce_read_comp() and p_cce_write_comp() treated float formats differently
from integer formats. The functions are identical now.
- The result of the internal float conversion check was ignored.
- Function p_rewrite_header() checks whether the header changes can coexist
with the data on disk. This check on the main header properties was too
relaxed: it only checked whether all non-modifiable fields were changed.
Not it checks whether any of these fields is changed.
- Function p_check_component_sizes() did not allow multiplexed extra
components.
- Function p_check color_format() did not check the multiplex factors.
This function (and p_create_free_header) now use a lookup table.
Functionality
- Function p_rewrite_header() was unnecessary strict in the allowed
modifications. Now also the following component properties may be changed:
com_code (i.e. component name), tem_sbsmpl, lin_sbsmpl, pix_sbsmpl,
tem_phshft, lin_phshft, pix_phshft.
- For compatibility with old non-cpfspd applications, it is possible to
override the default number (4) of file description records.
example: p_check_header( &header );
header.disable_hdr_checks = 1;
header.nr_fd_recs = 0;
p_write_header( filename, &header );
- New function p_mod_set_comp_2() replaces p_mod_set_comp() with the
additional component's multiplex factor (value 1 in virtually
all cases, but 2 for a multiplexed U/V component).
Function p_mod_set_comp() is translated automatically to p_mod_set_comp_2()
by a macro in cpfspd.h but it is considered obsolete and its use is
strongly discouraged.
- New function p_get_comp_2() replaces p_get_comp() with the
additional component's multiplex factor (value 1 in virtually
all cases, but 2 for a multiplexed U/V component).
Function p_get_comp() is translated automatically to p_get_comp_2()
by a macro in cpfspd.h but it is considered obsolete and its use is
strongly discouraged.
Miscellaneous
- Tests for convenience functions are added both for fileio and stdio.
Not all cases are checked though. Currently:
application buffers: float and double
file data formats: 8 bits and 16 bits
in all combinations.
- Minor update of the fio module in the speed test program: open mode
now checked for errors.
- Trimedia compiler release changed from tcs2.0f to tcs2.1-0088.
This release is also available on Linux.
- Copyright date updated.
Done in release 1.9.3 (dd. July 21, 2004)
-----------------------------------------
Highlights
Float data format.
2 bug fixes.
Bug fixes
- fixed in file cpfspd_hdr.c: passing NULL pointer as filename
results in a crash.
- fixed in file cpfspd_low.c: reopening a file that previously
failed to open (because it does not exist) results in a crash.
- fixed in cpfspd_low.c a bug with a test on the number of fd_recs
in a header.
Functionality
- Internal 16-bit float data format. This data format can be used to
store floating point numbers without application specific conversion
parameters. Valid range is 0, (+/-)[2^-14 .. (2^15 - 2^5)].
The system is assumed to have IEEE 754 single precision floats.
Use should be limited to extra components (e.g. motion vectors).
- Alignment of header data to 512 bytes. Included a
#define P_BYTES_REC in cpfspd.h and some more functionality in
src/cpfspd_hdr.c
Miscellaneous
- None
Done in release 1.9.2 (dd. Nov 25, 2003)
----------------------------------------
Highlights
Upgrade of MTK. Added functionality of reading components
into arbitrary buffers.
Bug fixes
- The component name returned by function p_get_comp() was not always
null terminated in case the component name has a length of 5 characters.
Functionality
- New convenience functions to read or write a component to an arbitrary
typed user buffer: p_cce_read_comp() and p_cce_write_comp().
In particular, these functions allow type conversion, shifting the
signal by an offset and scaling the singal by a gain parameter.
Typical use is a vector component with e.g. 2 bits subpixel precision
that is stored in a floating point application buffer.
- Function p_mod_rm_comp() performs no operation in case the
component id is -1. Previously, it reported an error.
This allows easy function nesting for applications:
stat = p_mod_rm_comp(header, p_get_comp_by_name(header, name));
Miscellaneous
- Upgrade to MTK version 5.2.1
+ New invoke script supports win32 platform.
+ Make.mtk supports the Microsoft Visual C/C++ compiler under cygwin.
+ Parse_options bugfix
- As a result of the new MTK invoke script, the test scripts
in test and speed can execute the win32 target under cygwin.
- As result of the new MTK, the library name on win32 is now
'cpfspd.lib'; the old names 'cpfspd_mt.lib' and 'cpfspd_st.lib'
are not used anymore. The newly build library has been tested
with single threading and multiple threading applications.
The files "make.bat", formerly used to compile and link
with the MS Visual C/C++ compiler are removed.
Done in release 1.9.1 (dd. June 13, 2003)
-----------------------------------------
Highlights
A severe bug causing an assert when opening a nonexisting file
is fixed. The CPU load is reduced on the win32 platform.
On a PC with 6 disks in RAID configuration, we are now able
to read 320 Mbyte/s (CPU load on 2.4GHz Xeon is 62%).
Bug fixes
- An attempt to open a nonexisting file resulted in an assert failure
with core dump on windows. This bug was introduced with the variable
buffer size.
Functionality
- None
Miscellaneous
- On computers with a high performance disk system we found that
that the CPU load was the limiting factor in the cpfspd performance.
This was observed on systems with 6 SCSI disks (two pairs of 3 disks
the three disks managed by a hardware raid controller and the pairs
setup for software raid). This cpu load is mainly caused by
a memcpy operation. On the win32 target, the standard memcpy
function is now replaced by a special version that uses MMX
instructions and handles cache prefetching. As a result, the
cpu load has dropped considerably:
release reading writing
cpu bandwidth cpu bandwidth
1.9.0 92% 294MB/s 52% 222MB/s
1.9.1 62% 323MB/s 48% 260MB/s
Note: due to compiler support for MMX assembly instructions,
this is implemented in the win32 target only (compiled with
Microsoft Visual C), not in the cygwin target.
- On MSwindows file I/O, a performance drawback is resolved. When
reading images sequentially, an I/O request is running asynchronously
at a buffer ahead of the current read buffer. Sometimes an "fseek()"
is carried out that jumps into the I/O buffer. In this case,
the I/O buffer was read twice (also resulting in an extra disk
seek operation). This situation is now recognized and the extra read
is avoided. Although this situation did occur only every now and then
(depending on image size and buffer size), this change may improve
the read performance measurably on high performance disk systems.
- Minimum width and height of images in 'speed' test application
are forced at 24x16 pixels. This avoids a core dump when invoking
the program for small images.
- A new section "Harddisk performance" is added in the Readme.txt file.
Done in release 1.9.0 (dd. May 28, 2003)
----------------------------------------
Highlights
This release has some extensions in its interface:
+ the number of components can be retrieved
+ the buffer size can be adapted for performance
tuning on win32 systems
Further, the library can now be combined with applications
build with other compilers (i.e. a different compiler
than used to build the library itself).
Finally, some small bugfixes are implemented.
Bug fixes
- When an application was compiled with the native OS compiler,
linking the cpfspd library was not possible due to a missing
reference to "__eprintf". This was caused by a hidden dependency
of the gcc compiler used to build the cpfspd lib. This happened
on HP, SGI and Sun.
- Opening more than 10 files simultaneously could result in a
core dump on some platforms (observed on linux and windows).
This was caused by an "fclose()" call with NULL pointer in function
p_get_file_pointer() of module cpfspd_low.c.
- Files opened for writing and closed because more than 10 files
were simultaneously open could have an incorrect length field in
the header. The close function did not update the length field.
This is fixed.
- Field 'disable_hdr_checks' is initialized to 0 (false, the default
behavior) in function p_read_header().
(Note that this was already done in functios p_create_ext_header()
and p_create_header()).
This avoids reading the value from uninitialized memory (which
may result in random behaviour depending on the value of such memory
location).
This also means that the order of operations has become significant.
Applications that use the field disable_hdr_checks shall set
this field after one of these functions! Such application
may need modification before they can be used with this cpfspd
release.
Functionality
- New function p_get_num_comps() is added. It returns the total
number of components described in the header.
- Two new functions to handle file I/O buffer size:
p_set_file_buf_size() and p_get_file_buf_size().
This allows performance optimization on the win32 platform.
See comments in the file for more explanation.
Miscellaneous
- Upgrade to mtk version 5.2.0
- Upgrade to compiler gcc version 3.2.
This fixed the native OS compiler linking bug.
- A new test script xcomp.sh is added to verify native OS compilation.
This compiles and links a small test program xcomp.c using the
native OS compiler (command "cc").
- A new test script openfiles.sh using program openfiles.c is added to
verify the behaviour with many file simultaneously open.
- The speed test program is enhanced by new command options:
-bufsize, -width and -height.
- New test script added to evaluate the effect of various buffer sizes:
speed/buffer.sh. This is currently only useful on win32 since the buffer
size is only used on this platform.
Done in release 1.8.2 (dd. December 9, 2002)
--------------------------------------------
Highlights
Bugfix release affecting mulitple open and close if a
file within a single program. Further, arbitrary pfspd
files are supported now by disabling normal checking
(fully at the risk of the programmer).
Bug fixes
- A file opened by cpfspd for writing could not be read.
This situation typically occured when a pfspd file is written,
and then read within the same program. This is fixed. Now
the file is automatically closed and reopened when a read
access is performed on a file open for writing.
- A file, written in a program, then closed and reopened with a
new header structure could not be extended by writing an image.
Functionality
- An extra field is added to the pT_header structure: disable_hdr_checks.
By default, this flag is 0. When set, function p_check_header()
returns always P_OK. This function is also used internally by others.
With checking disabled, and using the low level data access function
on single components, any pfspd file can be accessed. This flag
must be set by the application (no API support) and all risks of
incorrect use are with the application programmer.
Miscellaneous
- Section "purpose" in Readme.txt file is extended.
- A new section "Ease of use versus functionality" is added to
the Readme.txt file.
Done in release 1.8.1 (dd. September 18, 2002)
----------------------------------------------
Highlights
Bugfix release; affects
- the use of extra components
- interlaced files with odd frame height
Bug fixes
- Definition of data width for extra components can differ from file
data width. This was not handled correctly. Fixed now.
- The file speed/test.sh works again. It contained debug print
code followed by exit to view values of internal variables.
- Interlaced files with odd frame height were accepted
(when frame_height == 2 * field_height + 1).
For interlaced files, the component height in the header is
the field height; there is no way to define different field
heights in pfspd files.
As bug fix, this situation is recognized and an error is generated.
For progressive files, an odd frame height is still possible.
- A check on component size for extra component is added.
Also here, the component size must 'fit' the frame size.
In case of failure, a new error P_WRONG_EXTRA_COMP_SIZE is generated.
Functionality
- None
Miscellaneous
- Description of read_mode and write_mode of single component
functions is added in cpfspd.h
- Upgrade to mtk release 5.1.1 (no functionality impact).
Done in release 1.8.0 (dd. June 12, 2002)
-----------------------------------------
Highlights
- File description added
- Large files (>2Gbyte) possible
- Files with extra components can be handled
- Performance improvement of factor 2 to 8 on pc local disk
- Native 32 bit windows library and test added (i.e. non cygwin)
Performance
The table below shows disk performance measurements on some different
computers with the showing the improvements of the various measures
taken in this release. Each measurement accesses only a single file;
all numbers in Mbyte/s (one standard definition stream 8 bits data
requires 22 Mbyte/s).
The tests show:
- std: normal fwrite/fread calls on large (frame size) buffers
- raw: (only PC) unbuffered I/O on large (frame size) buffers
- old: performance of cpfspd release 1.7.0
- open: keep pfspd files opened till application exits
- unbuf: as open and use unbuffered I/O on 256k buffers
- async: as unbuf and use asynchronous I/O
The measurement of "async" shows the current cpfspd performance.
The measures "open", "unbuf" and "async" all give cumulative performance
gains on PC platforms. The measure "open" is also implemented on unix
systems. Especially when writing files, this gives significant performance
gain since the file length is only updated in the header when the
file is closed.
On PC's, we used the local file system with files of 4 Gbyte.
On the unix systems, we use the network file system /home/vdata
with a file size of 2 Gbyte.
std raw old open unbuf async
WR RD WR RD WR RD WR RD WR RD WR RD
Fast (1) 49 85 119 131 10.5 15.0 33 22 93 89 93 114
Home (2) 17.3 13.5 27.0 36.1 10.2 12.7 14.9 21.7 28.2 35.0 30.0 35.0
Nbook(3) 7.5 6.1 10.9 11.8 6.2 7.0 7.2 10.0 10.9 11.8 10.0 11.1
hpcs1 8.9 8.9 1.3 7.9 9.8 8.7
suncs1 11.1 9.8 4.2 8.0 9.6 9.5
yankees 4.8 7.0 3.3 1.4 15.3 4.9
linux20 6.2 9.2 3.5 7.7 6.4 9.6
Total gain:
write read
Fast 8.8x 7.6x
Home 1.8x 2.6x
Nbook 1.3x 1.8x
hpcs1 7.5x 1.1x
suncs1 2.3x 1.2x
yankees 4.6x 3.5x
linux20 1.8x 1.2x
1) Dual pentium xeon 1.7Ghz, striped volume of 3 scsi disks
2) 1.4 GHz AMD Athlon, maxtor 7200 rpm 60 Gbyte disk
3) 650MHz PIII, 16 Gbyte notebook disk
Bug fixes
- None
Functionality
- Pfspd files with extra components can be handled now.
The extra components (which are not part of the color specification)
must be at the end of the component list. So the "normal" components
still need to be the first ones in the normal order (e.g. Y, or Y, U/V,
or Y, U, V etc). In the past, such files were not accepted and an error
"Illegal file or color format" was generated. The existing data access
routines silently ignore the extra components.
As a result, the special treatment of the padding component
P_P_COM_CODE "P " is not required anymore.
- Read/write of extra components is supported now.
New functions for data access:
+ p_read_field_comp()
+ p_read_frame_comp()
+ p_write_field_comp()
+ p_write_frame_comp()
+ p_read_field_comp_16()
+ p_read_frame_comp_16()
+ p_write_field_comp_16()
+ p_write_frame_comp_16()
New functions for header control:
+ p_mod_add_comp()
+ p_get_comp_by_name()
+ p_mod_set_comp()
+ p_mod_rm_comp()
+ p_mod_rm_extra_comps()
+ p_get_comp()
+ p_get_comp_buffer_size()
New errors:
+ P_INVALID_COMPONENT
+ P_READ_INVALID_COMPONENT
+ P_WRITE_INVALID_COMPONENT
+ P_ILLEGAL_TEM_SBSMPL
Removed errors:
+ P_POSITIVE_SEEK_ON_STDOUT
- File description is supported; maximum size is 2kbyte.
The file description is considered part of the header and
is part of the pT_header structure. As usual, higher level
support functions are available in cpfspd to easily access
and modify the description.
Note that this change increases the size of the pT_header
structure to 2880 bytes.
When reading pfspd files with more than 2kbyte file description,
the remainder description data is silently ignored.
Detailed modifications:
In cpfspd.h:
+ P_SDESCRIPTION definition added, value 2048.
+ Field 'description' added in pT_header.
New functions:
+ char *p_get_file_description (const pT_header *header);
+ pT_status p_mod_file_description (pT_header *header, char *description);
+ pT_status p_rewrite_header (const char *filename, pT_header *header);
New errors:
+ P_EXCEEDING_DESCRIPTION_SIZE
+ P_REWRITE_ON_STDOUT
+ P_REWRITE_MODIFIED_HEADER
Removed error:
+ P_NO_FD_RECORDS_ALLOWED
As part of this change, all files written with cpfspd will
allocate 2k description space in the header without any
modification of application programs. This allows
easy modification/addition of a file description.
Obviously, old files without file description can still be read.
Note that existing files may nog have the space available for the
description. In that case, copy the file, e.g. using pfspd_cat
and then modify the description.
- Large files (file size larger than 2Gbyte) are supported.
This requires platform dependent conditional compilation.
When nothing special is defined in the makefiles, normal ANSI-C
compilation is performed. On the supported platforms for
this release, large file support is available. New module:
cpfspd_fio.c to handle the low level file I/O. Modifications
only in module cpfspd_low.
Note that even if the software supports large files, the
actual operation may still depend on other factors, such as
limits in the operating system or the file system. So this
does depend on local configuration of computer and network
systems.
Large files are not supported in Windows 95, Windows 98 and
Windows ME. Support is available in Windows NT, Windows 2000
and Windows XP. Large files have been tested on the all platforms
using Nat. Lab. computers. Only on Linux, large files do not
work via the batch queue mechanism. Direct "rsh" access is
required. This will be fixed in a next release of the queing
system.
Note: depending on the platform, it may not be possible to
pipe large files through stdin/stdout. Therefore, this feature
is currently not part of the test script.
- The strategy how to handle file length is changed.
Previously, the file length was adapted at each image,
in case a new image was added to the file (growing the file
larger than the header indicates). This allowed an other tool
to look at the new image immediately.
Currently, the file length is only adapted when the file is
closed. As a result, the other tool will not be able to
"see" the new image until the file is closed.
Reason: significant speed improvement when writing data to disk.
Miscellaneous
- For performance improvement, cpfspd now opens a files at
its first access and keeps the file open. At most 10 files
can be open simultaneously. If more files are accessed by
the application, the least recently used one is closed and
the new one opened. When the application exits, all open
files are closed automatically. Normally, this is completely
transparent to the application. For exceptional cases where
detailed control is requied, function p_close_file() is added.
In multi-threading applications, p_open_file() may be required.
- For further performance improvement on the PC platform,
unbuffered and asynchronous I/O is used. The cpfspd library
keeps track of buffering itself. This results in a significant
performance improvement. The new test program "speed" shows that
cpfspd is faster than standard fread/fwrite and close to the
theoretical maximum of raw I/O.
- New section "File open/close and file length" in the readme file.
- A description of multi-threading cpfspd useage is added to the
readme file.
- Native 32 bit Windows library and test application are added
to the release kit now. This is achieved by DOS scripts
"make.bat" in both src directories. The library is stored in
the ".../cpfspd/lib/win32/cpfspd.lib" file. To build the magic
number, a DOS version of the CKSUM command is required.
See the library make.bat file. The cygwin test script can
also test the win32 executable: "test.sh win32".
This library compilation takes care of proper magic number
generation and large file support.
- A new test application, called "speed" is added in directory
.../cpfpsd/speed. This application measures the performance
of a file system. It can run in three modes:
+ raw (unbuffered) disk I/O
+ std (normal buffered) disk I/O
+ yuv performing 8 bits 4:2:2 standard definition pfspd file I/O
This application has been used to measure the performance
improvements of cpfspd.
- Upgrade to mtk release 5.0.4.
+ Simplified makefile structure.
+ Revised setup structure.
- CVS id now part of standard header.
Done in release 1.7.0 (dd. September 18, 2001)
----------------------------------------------
Highlights
- Cross compilation on TriMedia is now possible.
Precompiled libraries in big and little endian
mode are available in the release.
- The low level read and write routines are speed optimized.
- For data formats > 8 bits, both big and little
endian files can be read. In the past, a 10 bits
file, created on Linux with pfspd-al could not be read.
Bug fixes
- File cpfspd_low.c, function p_write_hdr() contained a bug.
The file is filled with zero image data. In case format is NOT
P_8_BIT_FILE, and record length > 1024 this filling was incorrect.
This might cause an crash (access violation), but we have never seen
that happening. The bug is fixed.
Functionality
- There is a new test on the opening of a header to check if the file
is a pfspd file. On all fields of the file header and component
headers, the syntax is checked. In all practical cases, this
avoids reading non-pfspd files. In the past, parsing header
information from an arbitrary text file could result in weird
header values causing an application crash.
- Cpfspd is now able to read little endian files. Such files can be
created on little endian CPU's (e.g. Linux using an Intel CPU)
with other packages than cpfspd (e.g. pfspd-al). Since endian
mode relates to byte ordering of a data word, this applies only
to files with data size > 8 bits. Cpfspd writes always big endian
files. For this purpose, field "little_endian" is added to
struct pT_header, defined in "cpfspd.h". Applications usually
do not need modification. Routine p_print_header() prints
this information.
Miscellaneous
- Upgrade to MTK version 4.0.0. This allows:
+ Cross compilation on TriMedia.
+ Some minor changes in cpfspd_mag.c due to a modified magic number system.
+ Although it is not recommended, the code can now be compiled in a
different make system. As a result, the version identification and
magic number is not available. These features need to be implemented
by the other system.
- Cpfspd can write files with very high number of bytes per record;
this was limited to 2048. There is still a minimum of 64 (as
defined by the pfspd standard).
- Removed memcpy in case of 8 bit file and 8 bit memory.
This speeds up TriMedia simulation... (and other platforms).
- Low level read and write routines are speed optimized.
On various platforms all read and write routines operate
significantly faster (depending on file system performance):
cygwin: no difference
hp : 15%
sun : 6%
sgi : 5%
linux : 57% (!!!)
This is tested by running pfspd_convert without options on a 8bit
file with both input and output on the /tmp disk. Since pfspd_convert
uses 16 bits memory, both the read and write routines perform the
type conversion and shift.
- Add setup scripts to provide for default paths to TCS directory.
- Comment added in cpfspd.h: there shall always be a header structure
for each pfspd file accessed by the program.
- Unused parameters stream_debug and print_debug removed from
function p_write_hdr() in file pfspd_low.c.
- Readme.txt file from test dir removed; not required for new mtk anymore.
Done in release 1.6.2 (dd. July 23, 2001)
------------------------------------------
- Internal test release; not distributed; changes listed under rel. 1.7.0.
Done in release 1.6.1 (dd. July 17, 2001)
------------------------------------------
- Internal test release; not distributed; changes listed under rel. 1.7.0.
Done in release 1.6.0 (dd. March 22, 2001)
------------------------------------------
Bug fixes
- In cpfspd_low.h const is removed before int print_error in p_read_image().
The const gave rise to warnings in MS Visual C.
Functionality
- The functionality of p_mod_image_size() is slightly changed:
- It does not modify the line and pixel frequency anymore when the new
image size is smaller than the existing image size (since this does
not make any sense).
This way when one first creates a header than contains a _larger_
image than the target image size, the modified image size can be
used on the video simulator to display the sequence using the larger
raster (the simulator will add black data around the smaller image).
- When the new image size is larger than the existing image size all
line and pixel frequencies are set to _zero_.
- New function p_mod_defined_image_freq() added.
Miscellaneous
- Updated makefiles to mtk version 1.9. This uses a new platform convention
for cygwin: regardless of the host OS it will use cygwin/ as the target
directory for all compilation results. This removes the need to link to
all possible cygwin uname version that exist (and that will change when
a new windows version arises). As a result all applications that use
the new version of the library should also upgrade to mtk version 1.9
(and also use the new invoke script, which is now included in the mtk).
- Readme.txt updated to new format.
- Updated makefiles to mtk version 2.0.0-indev.
Done in release 1.5.0 (dd. November 27, 2000)
----------------------------------------------
- Updated makefile to mtk version 1.7.
- Function added: p_fatal_error_fileio(). This function accepts a filename
string, which will be printed along with the error message on a fatal error.
- Function added: p_mod_defined_image_size(). This function changes the image
size to one of the defined image sizes of type pT_size.
- Implemented the possibility to read only selected components from
file (e.g. only B from a RGB file).
- Implemented checking of the file data format (like "B*8 " or "B*10"). Note
that the file data formats of all components should be equal.
- Added C structure to cpfspd.h to describe the file data format (from
P_8_BIT_FILE to P_16_BIT_FILE).
- Implemented new functions to read/write to/from buffers of type unsigned
short (function names end with _16). This means that also pfspd files of
types B*10, B*12, B*14 (10, 12, 14 bits per pixel) and I*2 (16 bits per
pixel) are supported.
- Implemented new function to get and modify the file data format:
p_get_file_data_format() & p_mod_file_data_format().
- The test programs are restructured.
Done in release 1.4.2 (dated October 4, 2000)
---------------------------------------------
- Updated makefile to mtk version 1.5. This includes revised support
for the Cygwin (win32 platform) compiler (also compilation with -mno-cygwin
is supported, though it is now handled differently than in version 1.4.1.
Use make BUILD_OPT=-mno-cygwin to build without linking to the cygwin dll).
- Function declarations in headers and source files are made consistent
(removed unneeded 'const' declarations).
- Change the behaviour of read_color in all p_read_* functions:
0 = do not read U/V (read only Y)
1 = read U/V (and also read Y)
2 = read _only_ U/V (do _not_ read Y)
Note that read_color only influences the reading of YUV formats.
- Extra tests added for reading only luminance or chrominance from YUV files.
Done in release 1.4.1 (dd. September 1, 2000)
---------------------------------------------
- Error code descriptions are slightly improved in pfspd_error.c.
- Updated makefile to mtk version 1.3. This now includes better support
for the Cygwin (win32 platform) compiler (also compilation with -mno-cygwin
is supported).
Done in release 1.4 (dd. August 25, 2000)
-----------------------------------------
- The function p_get_s_buffer_size was missing from the library. This has
been fixed.
- The function p_mod_image_size is added in order to change the size of the
picture.
Note: it is impossible to correctly calculate a new pixel and line
frequency based on the new width & height. The new values of the pixel and
line frequency that are calculated in this function only reflect an
approximation of the real new pixel and line frequencies; the calculated
values are only valid when the blanking intervals scale with the same
factor as the active part of the image (which is very unlikely).
In order to have correct values of the line and pixel frequency create a
new header with p_create_ext_header in the correct format.
- It was found that reading/writing of the format P_COLOR_422_PL &
P_COLOR_420_PL was not implemented correctly in version 1.3 beta 2.
For these formats one probably wants to use a smaller stride for the U & V
buffers than the stride used for the Y buffer (since the width of the U & V
components is half of the width of the y component).
In order to achieve this, all planar read/write functions have been adapted.
A new parameter uv_stride has been added (note that this causes an
incompatibility with version 1.3 beta 2 of the library).
If the uv_stride is set to 0 (zero) the same stride is taken for all
(Y, U & V) buffers. Note that this is exactly the same behaviour as in
version 1.3 beta 2.
Also note that the uv_stride is ignored for the RGB format.
- Use mtk version 1.2 (make tool kit version 1.2, copied to source directory,
removed use of Make.env since it is not needed here).
Done in release 1.3-beta-2 (dd. June 22, 2000)
----------------------------------------------
- The functions p_read/write_frame/field_rgb have been removed. Their
functionality is now covered by p_read/write_frame/field_planar.
- Added definition of "S" streaming format. The "S" format is only
supported as frames. The functions p_read/write_frame are used for
reading and writing. Also the function get_s_buffer_size is added.
Functions p_create_[ext_]header are adapted to support the "S" format.
- Moved lint & qac error checking into main makefile.
- Changed readonly bit into modified bit. This way the new error checking on
an illegally modified header can be implemented in a way that is backwards
compatible with previous versions (version 1.2).
- Error checking on illegally modified headers is implemented.
- Added additional bookkeeping inside the header structure that is used
to protect against errors when reading from stdio, and to support skipping
of images when reading from stdio.
- Added test programs for typical use cases in test subdirectory
(tested & passed on HP-UX, IRIX64, Linux & CYGWIN_NT-5.0).
- Testing revealed a bug in the reading/writing of RGB files. This has been
fixed.
Done in release 1.3-beta-1 (dd. June 13, 2000)
----------------------------------------------
- Added possibility to use stdio. This way filters may be built
the can be used in unix pipes.
When the special filename "-" is used for a read function (e.g.
p_read_frame) data will be read from stdin.
When the special filename "-" is used for a write function (e.g.
p_write_frame) data will be written to stdout.
NOTE: stdio can only be used with some restrictions. The data
should be read/written sequentially, so first the header,
then image 1, image 2, and so on. When writing to stdout,
the header should contain the exact number of images to be
written.
N.B.: as of 1.3-beta-2 it is possible to skip images when reading
from stdio. This way reading only luminance from a stdin color
stream is supported.
- Added reading/writing of planar YUV files in 4:4:4, 4:2:2 & 4:2:0 format
(p_read/write_frame/field_planar).
- Added reading/writing of RGB files (p_read/write_frame/field_rgb).
- Added generation of several new SD/HD format headers (including CIF, QCIF,
all ATSC formats, function: p_create_ext_header).
- All header modification routines (p_mod_*) now can return error codes.
- Added several minor header manipulation functions (p_mod_aspect_ratio,
p_mod_to_interlaced, p_mod_to_onehalf_image_rate).
- Added several header feature extraction functions (p_is_progressive,
p_get_rgb_buffer_size, p_get_image_size, p_get_aspect_ratio).
- Added a copy header function (p_copy_header). Use of this function is
required when one wants to use a header that is read from a file as a
template for a new file that will be written to disk. Internally there
is bookkeeping in the header structure to prevent against errors, like
accidentally modifying a header that should not be modified anymore
(e.g. after a header structure has been written do disk).
N.B.: the header protection this is implemented in version 1.3-beta-2 does
require the use of p_copy_header. It is however recommended for
readability of the code.
- The file cpfspd.c is split into multiple c files.
- Use mtk version 1.1 (make tool kit version 1.1, copied to source directory,
removed use of Make.env since it is not needed here).
Done in release 1.2 (dd. April 05, 2000)
----------------------------------------
- Bug fixed in p_read_frame/p_write_frame when reading/writing fields
(incorrect use of field numbers).
- Bug fixed in p_check_header (incorrect use of strcmp).
- p_create_header changed (strncpy now also copies trailing '\0')
Done in release 1.1 (dd. March 31, 2000)
----------------------------------------
- New API with p_ prefix.
Done in release 1.0 (dd. somewhere in 1997)
--------------------------------------------
- Initial version, with only low level pfspd interface.
Download Latest Version
cpfspd-1.12.4.tgz (1.8 MB)
