From: Sam S. <sd...@gn...> - 2003-09-29 16:05:10
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-26 16:44:09 -0700]: > > For me the priority is to get the bulk of this stuff into cvs so I > can just go back to using current cvs and not have to worry about you > asking me to update the patch to be against a more recent version > than the one I'm using to develop it. in the last weeks I made almost no changes to the files you are working on. you can just do "cvs up" before sending the patches in and all my changes will be seamlessly integrated with yours. > Clearly there's more to do, but also clearly the rest can be done > starting from here. For instance, if you wanted the current version > to be consistent/working then it would be much more important to > update the non-unix code required for this change than to make gray > or buffered streams work. since you are not using non-unix platforms, you need not worry about those issues. just finish your side of the patch (buffered & gray) and you are done with it. let me be blunt and please do not take an insult: if the current version goes in, you will have all you want and will have no incentive to do buffered & gray, so they will never get done. > Sam Steingold writes: > > > > First, what's the advantage of always returning [same # values] > I guess this is not a good place to argue the issue. comp.lang.lisp is. > > first, I assume that you use emacs/d-mode.el. > !! never heard of it - is there some command like m-x c-mode ? > (I don't see d-mode.) > Where can I find doc on .d file format? clisp/src/CodingStyle. > > one cannot indent CPP directives like everyone else does. > I've only recently had significant experience with c - don't expect > me to understand c++ issues. CPP = C Pre-Processor (see "man cpp"), /= C++ > > > What's still on the critical path to this particular patch? > > 1. gray.lisp > > 2. buffered streams. > > I propose to make those into separate patches after this one. > I'll make all other changes above, verify that build still works > and put diffs in the usual place. > You put them in cvs, then I continue from cvs. > I hope additional patches will then be relatively small and painless. the current patches are already relatively small and painless. :-) > At very least, I suggest that this (C++) not be considered to be on > the critical path to the write-byte-sequence patch. OK. -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> I haven't lost my mind -- it's backed up on tape somewhere. |
From: Sam S. <sd...@gn...> - 2003-09-29 17:35:49
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-29 09:46:41 -0700]: > > Sam Steingold writes: > > > in the last weeks I made almost no changes to the files you are working > > on. you can just do "cvs up" before sending the patches in and all my > > changes will be seamlessly integrated with yours. > This suggests that you could have used my patches from 2.31 > which would have saved me significant trouble. come on: if there _was_ trouble, than someone _had_ to go through it. if you make it a habit (or just add to your crontab) to do "cvs up" daily, these problems will dissipate. trust me. > > let me be blunt and please do not take an insult: if the current > > version goes in, you will have all you want and will have no > > incentive to do buffered & gray, so they will never get done. > > Although I understand the motivation I really think this is unfair. I want the patch to be consistent and complete. "If you do it, do it right". > If you want them so badly perhaps you should do them. I will write the src/NEWS entry for you. :-) > That would be less work for you than even answering my questions and > checking the code I send you. I consider answering your questions a good investment. :-) > I've actually started to look at these cases. I think I do understand > the gray case but I don't think I understand the buffered case. please take a look at the READ-BYTE-SEQUENCE :NO-HANG patch. the output case should not be too different. (doing this will give you an opportunity to check that buffered sockets are faster than the unbuffered ones...) -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> Stupidity, like virtue, is its own reward. |
From: <don...@is...> - 2003-09-29 19:45:51
|
Sam Steingold writes: > I want the patch to be consistent and complete. > "If you do it, do it right". There are degrees of completeness. I argue that the current patch is a reasonable point. After that adding gray will be another and after that buffered streams will be another. After that there are many more, such as listen-write, and perhaps even trying to make no-hang work for bytes and then for write-sequence. I'm starting to look at the large file IO problem and it occurs to me that this is a good example of a patch that will not interleave well with the write-byte-sequence patch. For instance, all the lines that involve both new no_hang arguments and new types for #bytes will be affected by both. I guess you want me to finish the write-byte sequence patch before I start on large files. > I consider answering your questions a good investment. :-) speaking of which, give me an idea what this unfamiliar syntax is about? handle_lseek(stream,BufferedStream_channel(stream),position,SEEK_SET,); - parameter list ending with , unpack_sstring_alloca(*chararray_,len,start, charptr=); - variable followed by = > please take a look at the READ-BYTE-SEQUENCE :NO-HANG patch. > the output case should not be too different. > (doing this will give you an opportunity to check that buffered sockets > are faster than the unbuffered ones...) Where is this patch? |
From: Sam S. <sd...@gn...> - 2003-09-29 19:22:07
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-29 10:56:48 -0700]: > > Sam Steingold writes: > > I want the patch to be consistent and complete. > > "If you do it, do it right". > There are degrees of completeness. I argue that the current patch > is a reasonable point. After that adding gray will be another and > after that buffered streams will be another. After that there are > many more, such as listen-write, and perhaps even trying to make > no-hang work for bytes and then for write-sequence. you mean "chars", not "bytes". the problem with chars is fundamental: what to do when the i/o blocks inside a multi-byte character? keep it in a buffer? what about unbuffered streams? what if the end of the character never arrives? error? the problem with gray&unbuffered is a mere technicality. > I'm starting to look at the large file IO problem and it occurs to me > that this is a good example of a patch that will not interleave well > with the write-byte-sequence patch. For instance, all the lines that > involve both new no_hang arguments and new types for #bytes will be > affected by both. of course. > I guess you want me to finish the write-byte sequence patch before I > start on large files. absolutely. > > I consider answering your questions a good investment. :-) > speaking of which, give me an idea what this unfamiliar syntax is about? > handle_lseek(stream,BufferedStream_channel(stream),position,SEEK_SET,); > - parameter list ending with , this is a macro. do "etags *.d *.c *.lisp" in clisp/src, add clisp/src/ to your `tags-table-list' and use M-. (and M-* to get back) liberally. > unpack_sstring_alloca(*chararray_,len,start, charptr=); > - variable followed by = macro again. > > please take a look at the READ-BYTE-SEQUENCE :NO-HANG patch. > > the output case should not be too different. > > (doing this will give you an opportunity to check that buffered sockets > > are faster than the unbuffered ones...) > Where is this patch? C-x v l will give your the CVS log for the file. find the revisions are interested in as a region and "d" will give you the patch. all praise cvs & emacs! PS. you can do the same using the web interface to the CVS. -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> Modern man is the missing link between apes and human beings. |
From: <don...@is...> - 2003-09-29 20:19:58
|
Ah, mail to clisp-devel working again... Sam Steingold writes: > the problem with chars is fundamental: what to do when the i/o blocks > inside a multi-byte character? I think it's pretty clear - you have to check whether you can read/write enough for the next char. If not you don't read/write anything. > keep it in a buffer? what about unbuffered streams? independent of buffered-p > what if the end of the character never arrives? error? no, you just return without reading/writing that char This seems more of a problem for the hang case than the no-hang case. > > I guess you want me to finish the write-byte sequence patch before I > > start on large files. > absolutely. You are a hard taskmaster. > > - parameter list ending with , > this is a macro. which doesn't answer the question: what does it mean? Is this standard c that I can expect to find in K&R ? Where should I look for the answer? > > > please take a look at the READ-BYTE-SEQUENCE :NO-HANG patch. > > Where is this patch? > C-x v l will give your the CVS log for the file. undefined - I must need something for cvs I eventually found the patch in cvs browsing at sourceforge but - it's only for one file at a time - how do I find all the files that were affected? - it shows up in a 2 column format that I find hard to read (as in hard to see what the changes are) Is there a way to change the format? |
From: Sam S. <sd...@gn...> - 2003-09-29 21:29:01
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-29 13:14:05 -0700]: > > > > I guess you want me to finish the write-byte sequence patch before I > > > start on large files. > > absolutely. > You are a hard taskmaster. CLISP development has certain traditions. I learned them by submitting the same patch to Bruno several times until he was happy. I learned his ways and it was much easier afterward. > > > - parameter list ending with , > > this is a macro. > which doesn't answer the question: what does it mean? CPP works by literal substitution: #define FOO(a,b) a foo(b+1) then FOO(z=,d[3]); will expand to z=foo(d[3]+1); while FOO(,zoo()); into foo(zoo()+1); > Is this standard c that I can expect to find in K&R ? not sure - and judging by the existence of things like _EMA_ (empty macro argument) - grep the sources (and utils/) for it - unlikely. > Where should I look for the answer? C FAQ? > > > > please take a look at the READ-BYTE-SEQUENCE :NO-HANG patch. > > > Where is this patch? > > C-x v l will give your the CVS log for the file. > undefined - I must need something for cvs read Emacs manual (Version Control). > I eventually found the patch in cvs browsing at sourceforge but > - it's only for one file at a time - how do I find all the files > that were affected? ChangeLog. > - it shows up in a 2 column format that I find hard to read > (as in hard to see what the changes are) > Is there a way to change the format? cvs di -r ... -r ... will print the usual thing. -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> Are you smart enough to use Lisp? |
From: <don...@is...> - 2003-09-30 00:21:23
|
==== old business Sam Steingold writes: > CPP works by literal substitution: ah, of course ... > > Message has implicit destination > this means that you put clisp-devel into the Bcc field. Actually I think I had a blank to line, no bcc. still can't figure out what to do about this > > cvs [update aborted]: recv() from server cvs.sourceforge.net: EOF > there is an until in unixaux.d in a line you change. I guess you mean the diff has a + line with until in it. That same line also appears in a - line. ==== new business stream.d: /* UP: Positions a Byte-based File-Stream, so the next Byte can be read or overwritten. buffered_nextbyte(stream,no_hang) > stream : (open) Byte-based File-Stream. > no_hang: do not block < result : NULL if EOF else: Pointer to the next Byte changed in stream: index, endvalid, have_eof_p, buffstart */ local uintB* buffered_nextbyte (object stream, bool no_hang) { First, I've been wondering what UP: means. It's not yet clear to me whether endvalid points to the last valid or the first invalid. So far I think the first invalid. Anyhow, I don't see what's supposed to happen here if no_hang is true and the attempt to position the stream would block. It seems to me that in that case you would fail to satisfy the comment at the top (that the stream be positioned for next read/write) It looks to me like the function in that case returns 0 which is supposed to mean eof. This raises another issue in my mind. What's supposed to happen if you ask for non-blocking output to a file? I'd expect that if the block in which the data resides is not in the buffer then you return without writing anything. The question is whether that should then also initiate the operation to get the required data. If so, it's reasonable to try again later. If not then we have to do a blocking operation to get started, which doesn't seem desirable. It's not clear to me from man which of these (if either!) read/write does. |
From: <don...@is...> - 2003-09-30 05:50:53
|
oops, just realized that I hadn't sent this - was supposed to be sent before the last msg, which refers to it as "my arguments about buffered streams". Even when passed true for no_hang buffered_nextbyte can call buffered_flush which does not take a no_hang argument and in fact calls BufferedStreamLow_flush which in turn calls full_write. Are you sure you think buffered streams can support no_hang? |
From: Sam S. <sd...@gn...> - 2003-09-30 13:52:46
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-29 22:44:49 -0700]: > > oops, just realized that I hadn't sent this - was supposed to > be sent before the last msg, which refers to it as > "my arguments about buffered streams". > > Even when passed true for no_hang buffered_nextbyte can call > buffered_flush which does not take a no_hang argument and in fact > calls BufferedStreamLow_flush which in turn calls full_write. > > Are you sure you think buffered streams can support no_hang? You need to check that flushing will not block before actually doing it. -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> 186,000 Miles per Second. It's not just a good idea. IT'S THE LAW. |
From: <don...@is...> - 2003-09-30 07:13:35
|
well, I did manage to do cvs update, and for that I got a couple of mangled files: Merging differences between 1.379 and 1.380 into stream.d rcsmerge: warning: conflicts during merge cvspserver server: conflicts found in clisp/src/stream.d and Merging differences between 1.52 and 1.53 into unix.d rcsmerge: warning: conflicts during merge cvspserver server: conflicts found in clisp/src/unix.d I guess I'm supposed to find things like this <<<<<<< unix.d /* START_NO_BLOCK() & END_NO_BLOCK() should appear in pairs inside {NO_BLOCK_DECL() ...} */ #define NO_BLOCK_DECL() int non_blocking_io = 1; ======= /* START_NO_BLOCK() & END_NO_BLOCK() should appear in pairs inside { NO_BLOCK_DECL(); ... }; NO_BLOCK_DECL() should be before the first statement, but after the last declaration */ >>>>>>> 1.53 and, if they seem sufficiently similar, throw out the top one along with the separator lines. I'm assuming your versions are all better but wondering about the interesting transformations like if ( fcntl(handle,F_SETFL,fcntl_flags) <0) { OS_error(); } => do{if (fcntl(handle,F_SETFL,fcntl_flags)<0) {OS_error();}}while(0) I don't understand why NO_BLOCK_DECL now seems to take an argument, but I've updated my uses of it. Is output from this command ok? cvs -z3 -d:pserver:ano...@cv...:/cvsroot/clisp diff -u clisp [oh, no - now getting more of these ... cvs [diff aborted]: recv() from server cvs.sourceforge.net: EOF ] |
From: Sam S. <sd...@gn...> - 2003-09-30 13:47:04
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-30 00:07:36 -0700]: > > well, I did manage to do cvs update, and for that I got a couple of > mangled files: > Merging differences between 1.379 and 1.380 into stream.d > rcsmerge: warning: conflicts during merge > cvspserver server: conflicts found in clisp/src/stream.d > and > Merging differences between 1.52 and 1.53 into unix.d > rcsmerge: warning: conflicts during merge > cvspserver server: conflicts found in clisp/src/unix.d these are the hunks in your patch that failed against the CVS, as I mentioned in my previous e-mail. > I guess I'm supposed to find things like this > <<<<<<< unix.d > /* START_NO_BLOCK() & END_NO_BLOCK() should appear in pairs inside > {NO_BLOCK_DECL() ...} */ > #define NO_BLOCK_DECL() int non_blocking_io = 1; > ======= > /* START_NO_BLOCK() & END_NO_BLOCK() should appear in pairs > inside { NO_BLOCK_DECL(); ... }; > NO_BLOCK_DECL() should be before the first statement, > but after the last declaration */ > >>>>>>> 1.53 > and, if they seem sufficiently similar, throw out the top one along > with the separator lines. you should have `reverted' unix.d and undid the failing hunk in stream.d > I'm assuming your versions are all better but wondering about the > interesting transformations like > if ( fcntl(handle,F_SETFL,fcntl_flags) <0) { OS_error(); } > => > do{if (fcntl(handle,F_SETFL,fcntl_flags)<0) {OS_error();}}while(0) been through it. see C FAQ question 10.4 > > I don't understand why NO_BLOCK_DECL now seems to take an argument, > but I've updated my uses of it. > > Is output from this command ok? > cvs -z3 -d:pserver:ano...@cv...:/cvsroot/clisp diff -u clisp > > [oh, no - now getting more of these ... > cvs [diff aborted]: recv() from server cvs.sourceforge.net: EOF > ] alas. please complain to the SF admins. they promised to fix this by Sept 1 and then by Oct 1. try $ until cvs up; do sleep 20; done and go get some sleep :-) -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> Why do we want intelligent terminals when there are so many stupid users? |
From: <don...@is...> - 2003-09-30 16:42:20
|
Sam Steingold writes: > you should have `reverted' unix.d and undid the failing hunk in stream.d But there's still a diff: -extern RETRWTYPE full_write (int fd, WRITE_CONST RW_BUF_T buf, RW_SIZE_T nbyte); +#define safe_write(f,b,n) write_helper(f,b,n,true) +#define full_write(f,b,n) write_helper(f,b,n,false) > > if ( fcntl(handle,F_SETFL,fcntl_flags) <0) { OS_error(); } > > => > > do{if (fcntl(handle,F_SETFL,fcntl_flags)<0) {OS_error();}}while(0) > been through it. > see C FAQ question 10.4 Where's this C FAQ ? > > Is output from this command ok? > > cvs -z3 -d:pserver:ano...@cv...:/cvsroot/clisp diff -u clisp Still want to know whether that form of output will be ok as patch for you. ==== > > oops, just realized that I hadn't sent this - was supposed to > > be sent before the last msg, which refers to it as > > "my arguments about buffered streams". > > > > Even when passed true for no_hang buffered_nextbyte can call > > buffered_flush which does not take a no_hang argument and in fact > > calls BufferedStreamLow_flush which in turn calls full_write. > > > > Are you sure you think buffered streams can support no_hang? > > You need to check that flushing will not block before actually doing it. I don't think that can be done. Flushing is not an atomic operation. You can get part way through it before you find that you'll block. The best you could do is give flush a no hang argument that would partially flush and report back that the rest would hang. In fact, I think this is already a bug in read_byte_array_buffered: It starts by doing buffered_nextbyte(stream,no_hang) which again may do a flush which can hang. Also, shouldn't read_helper be putting the file in non-blocking mode like write_helper does? Also, now that I look at it, #if (defined(GENERATIONAL_GC) && defined(SPVW_MIXED)) || defined(SELFMADE_MMAP) # Must adjust the memory permissions before calling read(). Does this stuff also apply to write_helper? I think there's an interesting question of what you think no_hang should do in the case where the data is available but just not immediately (as in waiting for a disk). And I think this depends on what the low level OS calls do. My impression is that in non-blocking mode when you do a read/write it can return 0 if it needs to access the disk, but in that case it does access the disk so that at some time later the same call to read/write will not return 0. [anyone out there really know the answer?] If that's true then it's reasonable to write loops with only calls to nonblocking operations - which I think is highly desirable. And I would like that to still be the case in lisp. I think in order to make this work for your buffered code you'd have to record how much of the buffer has been flushed so you could restart an operation (like read-byte-sequence) and have it continue to flush the buffer where it left off. If you don't do that then you'll never make progress if the buffer is too big for a single disk access. I propose to make no_hang input for buffered streams generate errors for now, just like it does for output, rather than trying to fix them both. |
From: Sam S. <sd...@gn...> - 2003-09-30 16:54:04
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-30 09:36:16 -0700]: > > Sam Steingold writes: > > > you should have `reverted' unix.d and undid the failing hunk in stream.d > But there's still a diff: > -extern RETRWTYPE full_write (int fd, WRITE_CONST RW_BUF_T buf, RW_SIZE_T nbyte); > +#define safe_write(f,b,n) write_helper(f,b,n,true) > +#define full_write(f,b,n) write_helper(f,b,n,false) indeed. > > > if ( fcntl(handle,F_SETFL,fcntl_flags) <0) { OS_error(); } > > > => > > > do{if (fcntl(handle,F_SETFL,fcntl_flags)<0) {OS_error();}}while(0) > > been through it. > > see C FAQ question 10.4 > Where's this C FAQ ? google --> <http://www.faqs.org/faqs/C-faq/faq/> > > > Is output from this command ok? > > > cvs -z3 -d:pserver:ano...@cv...:/cvsroot/clisp diff -u clisp > Still want to know whether that form of output will be ok as patch for you. yes, and please add "-w" to "-u" to drop whitespace-only changes. > > You need to check that flushing will not block before actually doing it. > > I don't think that can be done. Flushing is not an atomic operation. > You can get part way through it before you find that you'll block. > The best you could do is give flush a no hang argument that would > partially flush and report back that the rest would hang. sounds good. > In fact, I think this is already a bug in read_byte_array_buffered: > It starts by doing buffered_nextbyte(stream,no_hang) which again may > do a flush which can hang. you may be right. > Also, shouldn't read_helper be putting the file in non-blocking mode > like write_helper does? probably. > Also, now that I look at it, > #if (defined(GENERATIONAL_GC) && defined(SPVW_MIXED)) || defined(SELFMADE_MMAP) > # Must adjust the memory permissions before calling read(). > Does this stuff also apply to write_helper? No idea. Bruno? > If that's true then it's reasonable to write loops with only calls to > nonblocking operations - which I think is highly desirable. And I > would like that to still be the case in lisp. I think in order to > make this work for your buffered code you'd have to record how much of > the buffer has been flushed so you could restart an operation (like > read-byte-sequence) and have it continue to flush the buffer where it > left off. If you don't do that then you'll never make progress if the > buffer is too big for a single disk access. > > I propose to make no_hang input for buffered streams generate errors > for now, just like it does for output, rather than trying to fix them > both. No, please make no_hang work for buffered streams for both input and output _modulo_ the future fix of the buffered_nextbyte() bug that you just discovered. i.e., get everything done _assuming_ buffered_nextbyte() works as advertised, get the patch in, then work on buffered_nextbyte()/flush as the next iteration. (I think it would be better to nail buffered_nextbyte/flush before you move on to large files, but it is up to you). -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> When we write programs that "learn", it turns out we do and they don't. |
From: <don...@is...> - 2003-09-30 18:24:58
|
Sam Steingold writes: > google --> <http://www.faqs.org/faqs/C-faq/faq/> Excellent! Thanks, will read it. > yes, and please add "-w" to "-u" to drop whitespace-only changes. I hope there aren't any. (but might find out by doing a diff with and without) > > Also, shouldn't read_helper be putting the file in non-blocking mode > > like write_helper does? > probably. I suppose that should be a separate patch. I'll leave it to you. I already have a number of things in my patch that fix old doc for fns that now use no_hang. I guess I should have submitted those separately it doesn't seem worth the trouble now. > > Also, now that I look at it, > > #if (defined(GENERATIONAL_GC) && defined(SPVW_MIXED)) || defined(SELFMADE_MMAP) > > # Must adjust the memory permissions before calling read(). > > Does this stuff also apply to write_helper? > > No idea. > Bruno? [still hoping someone will know the answer to this: > > If that's true then it's reasonable to write loops with only calls to > > nonblocking operations - which I think is highly desirable. > > I propose to make no_hang input for buffered streams generate errors > > for now, just like it does for output, rather than trying to fix them > > both. > No, please make no_hang work for buffered streams for both input and > output _modulo_ the future fix of the buffered_nextbyte() bug that you > just discovered. So for now buffered streams will work with no-hang but they might hang. > i.e., get everything done _assuming_ buffered_nextbyte() works as > advertised, get the patch in, then work on buffered_nextbyte()/flush as > the next iteration. There's still my other complaint about buffered_nextbyte, that it should be able to return a different value to indicate would-hang. Right now the hang case seems to be interpreted as eof, which is not good. Then, of course, the callers with no_hang (I guess currently only one, but soon to be two) must look for that value. What do you think of this: /* UP: Positions a Byte-based File-Stream, so the next Byte can be read or overwritten. buffered_nextbyte(stream,no_hang) > stream : (open) Byte-based File-Stream. > no_hang: do not block < result : 0 = EOF (the next byte can be written, not read) -1 = would block (only if no_hang, next byte cannot yet be read or written) else Pointer to the next Byte (can be read or written) changed in stream: index, endvalid, have_eof_p, buffstart */ local uintB* read_byte_array_buffered (object stream, uintB* byteptr, uintL len, bool no_hang) { do { var uintB* ptr = buffered_nextbyte(stream,no_hang); if (ptr == (uintB*)NULL) => if (ptr <= 0) break; (Previous code didn't really need to distinguish between hang and eof.) local const uintB* write_byte_array_buffered (object stream, const uintB* byteptr, uintL len, bool no_hang) { /* *** no_hang not yet supported here */ var uintL remaining = len; var uintB* ptr; do { # still remaining>0 Bytes to be filed. ptr = buffered_nextbyte(stream,false); [change that false to no_hang] if (ptr == (uintB*)NULL) [I guess I'd change the NULL to 0 now ...] goto eof_reached; [insert] if (ptr < 0) return byteptr; > (I think it would be better to nail buffered_nextbyte/flush before you > move on to large files, but it is up to you). My window of opportunity to work on this stuff may be about over. |
From: <don...@is...> - 2003-09-30 22:46:42
|
> > > Also, shouldn't read_helper be putting the file in non-blocking mode > > > like write_helper does? > > probably. > I suppose that should be a separate patch. I'll leave it to you. no, I have to do it or I can't test nonblocking reading. I don't think you're justified in read_helper returning after the first read in the partial case (which I've renamed no_hang to make it like all the others). Instead I think you have to loop and test for EAGAIN as I did in write_helper. This also allows us to do what I need for buffered_nextbyte - distinguish between eof and block. So next question - how do you like to return this extra value? I guess the easiest way would be a global variable that tells whether the last one blocked. This won't work for multiple threads, will it? (How does errno work for multiple threads, or the other clisp variables for multiple values?) We could pass in a location to be written. (What's the normal way to do that?) > My window of opportunity to work on this stuff may be about over. (I imagine you all breathing a sigh of relief at this.) |
From: Sam S. <sd...@gn...> - 2003-10-01 00:56:25
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-30 15:40:35 -0700]: > > need for buffered_nextbyte - distinguish between eof and block. > > So next question - how do you like to return this extra value? > I guess the easiest way would be a global variable that tells whether > the last one blocked. This won't work for multiple threads, will it? > (How does errno work for multiple threads, #define errno *thread_specific_errno_address() > or the other clisp variables for multiple values?) they are thread-specific. > We could pass in a location to be written. (What's the normal way to > do that?) pass the address. > > My window of opportunity to work on this stuff may be about over. > (I imagine you all breathing a sigh of relief at this.) rather panic - if you are not done soon, you will abandon the patch... -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> If I had known that it was harmless, I would have killed it myself. |
From: <don...@is...> - 2003-10-01 05:05:17
|
After previous attempt with file streams I thought I might as well try buffered socket streams. You may recall that when I write-byte-sequence to an unbuffered socket stream with no-hang I see something like 58000 bytes written. When I use a buffered stream it's 100000, i.e., all of them. In hind sight, that's not surprising - the bug in flushing buffers as part of positioning for the next byte would cause that. (I also finally see some debugging output showing that wr_by_array_iau8_buffered was called.) However, I find that buffered input is not working. I suppose I should watch the packets to be sure, but I thought that telnet (which is at the other side of this connection) sent packets when I type enter. In any case, the behavior I see is consistent with that but suggests some other problems. The first time I do read-byte-sequence with no-hang I get the first line. After that I get zero bytes. Actually, I tried a line of length 100000 and in that case I got back values of 57194 and 42808 (two extra for the crlf). At first I thought this must be bugs in my new code, but I then found that - read-byte-sequence without no-hang never returns at all - read-byte never returns - moving to clisp version 2.26 read-byte still doesn't return - in fact socket-status also hangs I conclude that there's something seriously wrong with buffered socket streams. I don't feel obligated to make that work as part of the write-byte-sequence patch. BTW, I now see the cvs message about Bruno's large file patch, but I still don't see it in sourceforge cvs browsing. I'm worried that that patch is going to cause serious merging difficulties with w-b-s, which is why I was delaying it. Right now I seem stuck in the middle of trying to make the last changes Sam asked for. I have no confidence that I'm close to a reasonable solution or getting any closer. On the other hand, the code that's running now seems to work no worse than the previous code. I suppose you could claim it works better since at least buffered file io and buffered socket output seem to work with no-hang. I'd like to wrap this up. Can I post the current diff? So far I don't see Bruno's patch in cvs, and I want to send the diffs before I do. Right now I have a few debugging prints and notes in // comments. Also have not changed ChangeLog. I hope that's ok. http://don-eve.dyndns.org/w-b-s.patch4 |
From: Sam S. <sd...@gn...> - 2003-10-01 16:15:31
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-30 21:59:07 -0700]: > > I conclude that there's something seriously wrong with buffered socket > streams. I don't feel obligated to make that work as part of the > write-byte-sequence patch. of course. > I'd like to wrap this up. Can I post the current diff? > So far I don't see Bruno's patch in cvs, and I want to send > the diffs before I do. that's because the anon cvs is run from the backup server with a 24 hour delay. please do complain to the SF admins! > Right now I have a few debugging prints and notes in // comments. > Also have not changed ChangeLog. 1. if you name your build directories "build-*", they will not be mentioned in the "cvs diff" messages with "?". 2. Please send the ChangeLog entry ASAP, and I will commit your patch. Note that I will probably change some formatting, so your "cvs up" will not necessarily be clean (especially considering Bruno's patch...) > http://don-eve.dyndns.org/w-b-s.patch4 thanks! -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> Trespassers will be shot. Survivors will be SHOT AGAIN! |
From: <don...@is...> - 2003-10-01 17:19:54
|
Sam Steingold writes: > that's because the anon cvs is run from the backup server with a 24 > hour delay. please do complain to the SF admins! ok, have now sent a complaint about both that and cvs [diff aborted]: recv() from server cvs.sourceforge.net: EOF > 1. if you name your build directories "build-*", they will not be > mentioned in the "cvs diff" messages with "?". Thanks. Maybe all this stuff should be collected into an introductory guide for clisp developers. (If you tell me now that this already exists I'll scream.) Chapt 32 of clispnotes is the closest I see. If I ever recover from this experience I'll try to submit something to be added to that section. Perhaps to be pointed to from faq. > 2. Please send the ChangeLog entry ASAP, and I will commit your > patch. Note that I will probably change some formatting, so your I hope you'll also try to do something about the //'s and the ***'s. At least eventually. > "cvs up" will not necessarily be clean (especially considering > Bruno's patch...) You did once volunteer to supply this. I suppose there are standards here of which I'm unaware, just like everywhere else. Here's a starting point which you may alter (or discard) as you see fit. add :no-hang argument to ext:write-byte-sequence, also gray:stream-write-byte-sequence Major changes to stream.d, lesser changes to sequence.d, unixaux.d minor changes to genclisph.d, gray.lisp, lispbibl.d, subr.d, subrkw.d, unix.d [Should the affected functions be listed?] [Should I include remaining problems?] - no-hang with buffered streams (buffered_flush can hang) - input from buffered socket streams - not clear that NO_BLOCK stuff is working as desired ... I see also a need to patch impnotes. I'll be happy to send suggestions for that later if you wish. |
From: <don...@is...> - 2003-10-01 17:47:25
|
> [Should I include remaining problems?] > - no-hang with buffered streams (buffered_flush can hang) > - input from buffered socket streams > - not clear that NO_BLOCK stuff is working as desired > ... One more big one comes to mind - support for other platforms. How is this to be handled? |
From: <don...@is...> - 2003-10-01 01:33:27
|
I've put the NO_BLOCK calls into read_helper but I don't see any blocking in file streams. I suspect that this is not doing what I expected/wanted, but perhaps someone out there has a better explanation. I open a large file for reading (one that hasn't been opened for a long time, so it's not in ram): (setf st (open "/tmp/mailtranscript4" :element-type '(unsigned-byte 8) :buffered nil)) I use both buffered t and nil. I then do (time (loop for i below 40 do (print (read-byte-sequence a st :no-hang nil)))) Here I use both no-hang t and nil. The array length is 100000, so this reads 4MB, which I expected should be enough to block at least once. One reason for suspecting that non-blocking is not working is that I've not yet seen a return value other than 100000. The other is that the real time is consistently higher than run time by about a factor of 2. The results of time are below. I remove the space - always 10908 summary of real/run times buffered no-hang nil t nil 1.25/.72 1.5/.85 t 1.23/.68 1.58/.8-.9 At least one thing conclusion is that at least for this example (file streams on linux, just trying to read lots of data) unbuffered is faster than buffered. As I would have expected. raw data below buffered nil no-hang t Real time: 1.233856 sec. Run time: 0.68 sec. Real time: 1.45016 sec. Run time: 0.74 sec. GC: 1, GC time: 0.03 sec. buffered t no-hang t Real time: 1.652864 sec. Run time: 0.92 sec. Real time: 1.580222 sec. Run time: 0.89 sec. Real time: 1.582247 sec. Run time: 0.8 sec. buffered t no-hang nil Real time: 1.498612 sec. Run time: 0.85 sec. buffered nil no-hang nil Real time: 1.25297 sec. Run time: 0.72 sec. |
From: <don...@is...> - 2003-10-06 00:12:17
|
[looks like I never sent this] First, thanks for committing these. I'll try downloading current and see what happens for large files. On the issue about NO_BLOCK seeming not to work for file streams, I have a new theory. I had originally expected that in non-blocking mode read/write would treat disk access as a reason to block. I now think it does not. (I still think it should!) I think the reason is that paging is not considered a reason to block and all this file access (I'm guessing) is done by mmap, which makes it all look like paging. On the subject of default buffered for open: Perhaps it would be worth testing buffered/unbuffered IO speed for small operations on file streams. I imagine the OS is doing its own buffering so that read/write might be pretty cheap. In any case, the extra cost of buffering for large operations is not excessive so this doesn't bother me. I do like the current default of unbuffered for socket streams. At least don't change that until the buffered case works. Also I notice (and appreciate!) that certain /proc files now default to unbuffered - which causes the IO to work! On the question of using errno - I noticed afterward the saving_errno macro (stream.d) which seems just what I wanted. My notes for c developers recommends looking at clisp.d - I really meant lispbibl.d, though I suppose clisp.d is also worth while. If you plan to do anything with these notes please add the reference to lispbibl.d. |
From: <don...@is...> - 2003-10-01 01:49:13
|
Sam Steingold writes: > > (How does errno work for multiple threads, > #define errno *thread_specific_errno_address() As you see from the previous message I have yet to actually test this stuff but my current implementation (hack?) is to just rely on errno. Is this code correct? How would you rather see it done? Can I do #define myerrno *thread_specific_errno_address() And if so, where should I put it? read_helper: ... { NO_BLOCK_DECL(fd); if (no_hang) START_NO_BLOCK(fd); while (nbyte!=0) { retval = read(fd,buf,nbyte); if (retval == 0) break; else if (retval < 0) { if (no_hang && (errno == EAGAIN)) { // *** signal blocking state reached -- just use errno? printf("read_helper - read blocked\n");// [never yet executed] goto end;} #ifdef EINTR if (errno != EINTR) #endif return retval; } else { buf += retval; done += (RW_SIZE_T)retval; nbyte -= (RW_SIZE_T)retval; [I was going to remove the next line but forgot so this shows that the single call to read is doing the whole 100KB] if (no_hang) break; } } end: if (no_hang) END_NO_BLOCK(fd); } [Will the end no block will overwrite errno? That's what I hoped to find out in the next print I suppose I could save errno and assign it back here.] if (errno == EAGAIN) printf("returning with block from read_helper\n");// return done; } Of course, the really ugly part of this solution is that you have to worry about what other code you execute between this and the later check. At the moment I think it's ok: buffered_nextbyte result = BufferedStreamLow_fill(stream)(stream,no_hang); if (errno == EAGAIN){ printf("buffered_nextbyte returning -1\n"); // return -1; /* hang case */ } local uintL low_fill_buffered_handle (object stream, bool no_hang) { var sintL result = 0; var Handle handle = TheHandle(BufferedStream_channel(stream)); if (!no_hang || ls_avail_p(listen_handle(handle,false,NULL))) { begin_system_call(); result = read_helper(handle,BufferedStream_buffer_address(stream,0), strm_buffered_bufflen,no_hang); end_system_call(); if (result<0) # error occurred? OS_filestream_error(stream); } return result; } local uintL low_fill_buffered_socket (object stream, bool no_hang) { var sintL result; /* *** doesn't use no_hang? */ SYSCALL(result,sock_read(TheSocket(BufferedStream_channel(stream)), BufferedStream_buffer_address(stream,0), strm_buffered_bufflen)); return result; } [I've not tried to update sock_read, can't test it] |
From: Sam S. <sd...@gn...> - 2003-10-01 15:38:53
|
> * Don Cohen <qba...@vf...3-vap.pbz> [2003-09-30 18:43:05 -0700]: > > Sam Steingold writes: > > > (How does errno work for multiple threads, > > #define errno *thread_specific_errno_address() > > As you see from the previous message I have yet to actually test this > stuff but my current implementation (hack?) is to just rely on errno. > Is this code correct? How would you rather see it done? > Can I do > #define myerrno *thread_specific_errno_address() > And if so, where should I put it? no, the system header that defines errno (be it errno.h or stdlib.h) will define it correctly for your platform. e.g., modules/bindings/glibc/linux.lisp has a lot of problems making errno available because of that. the flip side is that in C you do not have to worry about errno. > Of course, the really ugly part of this solution is that you have to > worry about what other code you execute between this and the later > check. At the moment I think it's ok: > > buffered_nextbyte > > result = BufferedStreamLow_fill(stream)(stream,no_hang); > if (errno == EAGAIN){ > printf("buffered_nextbyte returning -1\n"); // > return -1; /* hang case */ > } this looks kind of yuky. maybe BufferedStreamLow_fill could return -1 instead of relying on errno? Bruno? > local uintL low_fill_buffered_handle (object stream, bool no_hang) { > var sintL result = 0; > var Handle handle = TheHandle(BufferedStream_channel(stream)); > if (!no_hang || ls_avail_p(listen_handle(handle,false,NULL))) { > begin_system_call(); > result = read_helper(handle,BufferedStream_buffer_address(stream,0), > strm_buffered_bufflen,no_hang); > end_system_call(); > if (result<0) # error occurred? > OS_filestream_error(stream); > } > return result; > } > > local uintL low_fill_buffered_socket (object stream, bool no_hang) { > var sintL result; > /* *** doesn't use no_hang? */ > SYSCALL(result,sock_read(TheSocket(BufferedStream_channel(stream)), > BufferedStream_buffer_address(stream,0), > strm_buffered_bufflen)); > return result; > } -- Sam Steingold (http://www.podval.org/~sds) running w2k <http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/> <http://www.mideasttruth.com/> <http://www.honestreporting.com> If I had known that it was harmless, I would have killed it myself. |
From: <don...@is...> - 2003-09-29 19:45:53
|
note: I include clisp-devel in the cc but it won't work. Feel free to forward this to the list if you want it there. Sam Steingold writes: > in the last weeks I made almost no changes to the files you are working > on. you can just do "cvs up" before sending the patches in and all my > changes will be seamlessly integrated with yours. This suggests that you could have used my patches from 2.31 which would have saved me significant trouble. > let me be blunt and please do not take an insult: if the current version > goes in, you will have all you want and will have no incentive to do > buffered & gray, so they will never get done. Although I understand the motivation I really think this is unfair. If you want them so badly perhaps you should do them. That would be less work for you than even answering my questions and checking the code I send you. I've actually started to look at these cases. I think I do understand the gray case but I don't think I understand the buffered case. How about if I do gray and you do buffered? > > > first, I assume that you use emacs/d-mode.el. Aha! only in clisp src! And it defines d-mode. This should make things much easier, thanks. > > !! never heard of it - is there some command like m-x c-mode ? > > (I don't see d-mode.) > > Where can I find doc on .d file format? > clisp/src/CodingStyle. Thanks again. > the current patches are already relatively small and painless. :-) Not to me! |