Menu
▾
▴
state-threads-devel
|
From: Wesley W. T. <we...@te...> - 2002-01-20 21:30:57
|
Has anyone tried to use libst with 64bit large file support on a 32bit platform? I assume it won't work, but wanted to check. I notice that the read/write apis all use size_t instead of off_t, so I doubt they'll work. However, what I want to know is whether or not sticking them in a st_poll would work: There's that new errno value for if you use a 32bit call on a 64bit file and I am concerned that they could break even those st functions without offset paremeter. Has anyone gotten or failed to get this to work? If it doesn't work, should this be fixed? Before you say 'why would you need 64bit support for sockets', it's not for sockets... I want to have the equivalent of 'tail -f' on several files at the same time I'm watching sockets for incoming requests. These files could be >> 2Gb. Thanks. -- Wesley W. Terpstra <we...@te...> |
|
From: Lev W. <vl...@ne...> - 2002-01-21 20:15:28
|
Wesley W. Terpstra wrote:
> Has anyone tried to use libst with 64bit large file support on a 32bit
> platform?
>
> I assume it won't work, but wanted to check. I notice that the read/write
> apis all use size_t instead of off_t, so I doubt they'll work.
FreeBSD has the 64 bit large file support on 32 bit code. read/write
returns obvious ssize_t, and it is 32 bit in size. So particularly this
will work right.
> However,
> what I want to know is whether or not sticking them in a st_poll would work:
> There's that new errno value for if you use a 32bit call on a 64bit file
> and I am concerned that they could break even those st functions without
> offset paremeter.
>
> Has anyone gotten or failed to get this to work?
> If it doesn't work, should this be fixed?
>
> Before you say 'why would you need 64bit support for sockets', it's not for
> sockets... I want to have the equivalent of 'tail -f' on several files at
> the same time I'm watching sockets for incoming requests. These files could
> be >> 2Gb.
lseek(fd, 0, SEEK_END);
while(1) {
ssize_t rsize;
rsize = read(fd, buf, sizeof(buf));
if(rsize > 0) {
/* Do things */
} else {
usleep(250000);
}
}
This will work on 32-bit OS'es supporting 64-bit file systems. The off_t
parameter is being used only in lseek().
--
Lev Walkin
vl...@ne...
|
|
From: Wesley W. T. <we...@te...> - 2002-01-22 08:29:32
|
On Mon, Jan 21, 2002 at 12:14:19PM -0800, Lev Walkin wrote:
> Wesley W. Terpstra wrote:
> >Has anyone tried to use libst with 64bit large file support on a 32bit
> >platform?
> >
> >I assume it won't work, but wanted to check. I notice that the read/write
> >apis all use size_t instead of off_t, so I doubt they'll work.
>
> FreeBSD has the 64 bit large file support on 32 bit code. read/write
> returns obvious ssize_t, and it is 32 bit in size. So particularly this
> will work right.
What the heck. You're right: I just checked my headers, read/write use
ssize_t even in LFS mode. I suppose that makes sense -- you can't read/write
more than 2Gb at once even if LFS is enabled. Who'd want to anyways?
So the API in libst for read/write is fine. I still have concerns though.
> >Before you say 'why would you need 64bit support for sockets', it's not for
> >sockets... I want to have the equivalent of 'tail -f' on several files at
> >the same time I'm watching sockets for incoming requests. These files could
> >be >> 2Gb.
>
> lseek(fd, 0, SEEK_END);
> while(1) {
> ssize_t rsize;
> rsize = read(fd, buf, sizeof(buf));
> if(rsize > 0) {
> /* Do things */
> } else {
> usleep(250000);
> }
> }
>
> This will work on 32-bit OS'es supporting 64-bit file systems. The off_t
> parameter is being used only in lseek().
However, I want to use libst, so where you have read(...) I'd have to put
st_read(...). That way libst could context switch and work on other tasks
while I was waiting for data---the whole point of using it here.
Suppose I open the file with proper lfs support and do the lseek. This will
work as you describe above. However, consider this:
My program is compiled with _FILE_OFFSET_BITS=64 which makes my read
lfs-aware. Unfortunately, libst does not have this toggle when compiled.
Hence it's read function (which st_read will call) is not lfs aware and will
return EOVERFLOW when I call it (section 2.2.1.25 of the X/OPEN LFS spec
<http://ftp.sas.com/standards/large.file/x_open.20Mar96.html>).
On the other hand, maybe the spec means that I get EOVERFLOW if I hadn't
opened it as an lfs-fildes? Perhaps the read is the same and it checks some
flag on the fildes?
This is why I was wondering if I could use libst with an lfs system.
--
Wesley W. Terpstra <we...@te...>
|
|
From: Lev W. <vl...@ne...> - 2002-01-22 19:48:24
|
Wesley W. Terpstra wrote:
> On Mon, Jan 21, 2002 at 12:14:19PM -0800, Lev Walkin wrote:
>
>>Wesley W. Terpstra wrote:
>>
>>>Has anyone tried to use libst with 64bit large file support on a 32bit
>>>platform?
>>>
>>>I assume it won't work, but wanted to check. I notice that the read/write
>>>apis all use size_t instead of off_t, so I doubt they'll work.
>>>
>>FreeBSD has the 64 bit large file support on 32 bit code. read/write
>>returns obvious ssize_t, and it is 32 bit in size. So particularly this
>>will work right.
>>
>
> What the heck. You're right: I just checked my headers, read/write use
> ssize_t even in LFS mode. I suppose that makes sense -- you can't read/write
> more than 2Gb at once even if LFS is enabled. Who'd want to anyways?
>
> So the API in libst for read/write is fine. I still have concerns though.
>
>
>>>Before you say 'why would you need 64bit support for sockets', it's not for
>>>sockets... I want to have the equivalent of 'tail -f' on several files at
>>>the same time I'm watching sockets for incoming requests. These files could
>>>be >> 2Gb.
>>>
>>lseek(fd, 0, SEEK_END);
>>while(1) {
>> ssize_t rsize;
>> rsize = read(fd, buf, sizeof(buf));
>> if(rsize > 0) {
>> /* Do things */
>> } else {
>> usleep(250000);
>> }
>>}
>>
>>This will work on 32-bit OS'es supporting 64-bit file systems. The off_t
>>parameter is being used only in lseek().
>>
>
> However, I want to use libst, so where you have read(...) I'd have to put
> st_read(...). That way libst could context switch and work on other tasks
> while I was waiting for data---the whole point of using it here.
>
> Suppose I open the file with proper lfs support and do the lseek. This will
> work as you describe above. However, consider this:
>
> My program is compiled with _FILE_OFFSET_BITS=64 which makes my read
> lfs-aware. Unfortunately, libst does not have this toggle when compiled.
What you probably missing is what the hell is "lfs-aware".
_FILE_OFFSET_BITS=64 does not change the _read()_ call in any way, so
it should be compatible.
> Hence it's read function (which st_read will call) is not lfs aware and will
> return EOVERFLOW when I call it (section 2.2.1.25 of the X/OPEN LFS spec
> <http://ftp.sas.com/standards/large.file/x_open.20Mar96.html>).
>
> On the other hand, maybe the spec means that I get EOVERFLOW if I hadn't
> opened it as an lfs-fildes? Perhaps the read is the same and it checks some
> flag on the fildes?
>
> This is why I was wondering if I could use libst with an lfs system.
Probably, you won't use the st_read instead of read. Why do you need
to do it? st_read() will never return EWOULDBLOCK or EAGAIN on hardware
filesystems, so the thread-switch logic has no meaning on plain files.
--
Lev Walkin
vl...@ne...
|
|
From: Wesley W. T. <we...@te...> - 2002-01-22 20:51:56
|
On Tue, Jan 22, 2002 at 11:47:12AM -0800, Lev Walkin wrote:
> Wesley W. Terpstra wrote:
> >On Mon, Jan 21, 2002 at 12:14:19PM -0800, Lev Walkin wrote:
> >>Wesley W. Terpstra wrote:
> >>>Has anyone tried to use libst with 64bit large file support on a 32bit
> >>>platform?
> >>
> >>lseek(fd, 0, SEEK_END);
> >>while(1) {
> >> ssize_t rsize;
> >> rsize = read(fd, buf, sizeof(buf));
> >> if(rsize > 0) {
> >> /* Do things */
> >> } else {
> >> usleep(250000);
> >> }
> >>}
> >>
> >>This will work on 32-bit OS'es supporting 64-bit file systems. The off_t
> >>parameter is being used only in lseek().
> >
> >However, I want to use libst, so where you have read(...) I'd have to put
> >st_read(...). That way libst could context switch and work on other tasks
> >while I was waiting for data---the whole point of using it here.
> >
> >Suppose I open the file with proper lfs support and do the lseek. This will
> >work as you describe above. However, consider this:
> >
> >My program is compiled with _FILE_OFFSET_BITS=64 which makes my read
> >lfs-aware. Unfortunately, libst does not have this toggle when compiled.
>
> What you probably missing is what the hell is "lfs-aware".
> _FILE_OFFSET_BITS=64 does not change the _read()_ call in any way, so
> it should be compatible.
The X/OPEN standard strongly implies that the implementations can and should
redirect the IO functions when LFS is enabled. This same standard defines
how normal 32bit calls should fail on large files if they have not declared
that they know who to deal with lfs.
On linux, at least, pread is redirected as are many other IO functions. read
appears to be an exception, but that is surely not a portable assumption to
make.
features.h tests:
#if defined _FILE_OFFSET_BITS && _FILE_OFFSET_BITS == 64
# define __USE_FILE_OFFSET64 1
#endif
Then unistd.h does:
# ifndef __USE_FILE_OFFSET64
extern ssize_t pread (int __fd, void *__buf, size_t __nbytes, __off_t
__offset)
__THROW;
extern ssize_t pwrite (int __fd, __const void *__buf, size_t __n,
__off_t __offset) __THROW;
# else
# ifdef __REDIRECT
extern ssize_t __REDIRECT (pread, (int __fd, void *__buf, size_t __nbytes,
__off64_t __offset) __THROW,
pread64);
extern ssize_t __REDIRECT (pwrite, (int __fd, __const void *__buf,
size_t __nbytes, __off64_t __offset)
__THROW,
pwrite64);
# else
# define pread pread64
# define pwrite pwrite64
# endif
# endif
> Probably, you won't use the st_read instead of read. Why do you need
> to do it? st_read() will never return EWOULDBLOCK or EAGAIN on hardware
> filesystems, so the thread-switch logic has no meaning on plain files.
Ahh, I thought select(...) would not claim a file ready for input if it was
at EOF. However, a test program does not behave this way---as you indicated.
That's really too bad, it would have been an elegant way of following
several files at once.
I suppose I can use st_sleep to yield control and thus never let libst even
see the file descriptors for the lfs files. That will certainly resolve any
concerns I have.
Well, out of all this I have a workable solution---even if it is polling.
Thanks.
--
Wesley W. Terpstra <we...@te...>
|
Thanks for helping keep SourceForge clean.
X