Re: [ST-devel] libst & large files

[Wesley W. T. <we...@te...>]

On Tue, Jan 22, 2002 at 11:47:12AM -0800, Lev Walkin wrote:
> Wesley W. Terpstra wrote:
> >On Mon, Jan 21, 2002 at 12:14:19PM -0800, Lev Walkin wrote:
> >>Wesley W. Terpstra wrote:
> >>>Has anyone tried to use libst with 64bit large file support on a 32bit
> >>>platform?
> >>
> >>lseek(fd, 0, SEEK_END);
> >>while(1) {
> >>	ssize_t rsize;
> >>	rsize = read(fd, buf, sizeof(buf));
> >>	if(rsize > 0) {
> >>		/* Do things */
> >>	} else {
> >>		usleep(250000);
> >>	}
> >>}
> >>
> >>This will work on 32-bit OS'es supporting 64-bit file systems. The off_t 
> >>parameter is being used only in lseek().
> >
> >However, I want to use libst, so where you have read(...) I'd have to put
> >st_read(...). That way libst could context switch and work on other tasks
> >while I was waiting for data---the whole point of using it here.
> >
> >Suppose I open the file with proper lfs support and do the lseek. This will
> >work as you describe above. However, consider this:
> >
> >My program is compiled with _FILE_OFFSET_BITS=64 which makes my read
> >lfs-aware. Unfortunately, libst does not have this toggle when compiled.
> 
> What you probably missing is what the hell is "lfs-aware". 
> _FILE_OFFSET_BITS=64 does not change the _read()_ call in any way, so
> it should be compatible.

The X/OPEN standard strongly implies that the implementations can and should
redirect the IO functions when LFS is enabled. This same standard defines
how normal 32bit calls should fail on large files if they have not declared
that they know who to deal with lfs.

On linux, at least, pread is redirected as are many other IO functions. read
appears to be an exception, but that is surely not a portable assumption to
make.

features.h tests:
#if defined _FILE_OFFSET_BITS && _FILE_OFFSET_BITS == 64
# define __USE_FILE_OFFSET64    1
#endif

Then unistd.h does:
# ifndef __USE_FILE_OFFSET64
extern ssize_t pread (int __fd, void *__buf, size_t __nbytes, __off_t
__offset)
     __THROW;
extern ssize_t pwrite (int __fd, __const void *__buf, size_t __n,
                       __off_t __offset) __THROW;
# else
#  ifdef __REDIRECT
extern ssize_t __REDIRECT (pread, (int __fd, void *__buf, size_t __nbytes,
                                   __off64_t __offset) __THROW,
                           pread64);
extern ssize_t __REDIRECT (pwrite, (int __fd, __const void *__buf,
                                    size_t __nbytes, __off64_t __offset)
                           __THROW,
                        pwrite64);
#  else
#   define pread pread64
#   define pwrite pwrite64
#  endif
# endif

> Probably, you won't use the st_read instead of read. Why do you need
> to do it? st_read() will never return EWOULDBLOCK or EAGAIN on hardware
> filesystems, so the thread-switch logic has no meaning on plain files.

Ahh, I thought select(...) would not claim a file ready for input if it was
at EOF. However, a test program does not behave this way---as you indicated.
That's really too bad, it would have been an elegant way of following
several files at once.

I suppose I can use st_sleep to yield control and thus never let libst even
see the file descriptors for the lfs files. That will certainly resolve any
concerns I have.

Well, out of all this I have a workable solution---even if it is polling.

Thanks.

-- 
Wesley W. Terpstra <we...@te...>