From: Daniel S. <dv...@ea...> - 2009-11-24 15:33:42
|
Sam, Perhaps a dumb question. I should probably have learned this in some obvious reverence. I wrote a sound-generation program that makes .AIF files. I write the sound samples sequentionally into the file with this ... (defmacro write-integer (n outfil &optional (nbytes 2) &aux stash) (dotimes (i (1- nbytes)) (push (list 'ash n (* -8 (1+ i))) stash)) `(progn ,@(mapcar (lambda (x) `(write-byte (logand ,x 255) ,outfil)) (reverse (cons n stash))))) It writes integers of nbytes into a binary file. In order to do rescaling, I want to write binary files of floating-point numbers, ... and fast. Put differently, I want to #'logand and #'ash the bits of floating-point numbers. The CL floating-point conversion functions don’t quite do the job. They will break it up into <sign exponent mantissa>. I want something like #'float-as-integer, and then something to undo that on reading it back in. -- devious dan |
From: Sam S. <sd...@gn...> - 2009-11-24 16:24:40
|
Dan, Daniel Starr wrote: > > The CL floating-point conversion functions don’t quite do the job. > They will break it up into <sign exponent mantissa>. > I want something like #'float-as-integer, and then something to > undo that on reading it back in. write-float and read-float should do what you want. http://clisp.cons.org/impnotes/bin-io.html note that your macro write-integer should probably be dropped in favor of clisp's built-in write-integer Sam. |
From: Sam S. <sd...@gn...> - 2009-11-24 16:51:56
|
Fred Cohen wrote: > Since binary I/O is coming up, I have long wanted to be able to do all > of the operations I do on ASCII characters and sequences on bytes... > Somehow it would be nice if we could identify a character set (e.g., > ASCII / EBCDIC / BYTE / UNICODE / etc.) and have everything work in > that set instead of in what is something like ASCII-7 today. I am confused. We already have encodings. http://clisp.cons.org/impnotes/encoding.html Is that not what you want? |
From: Fred C. <fc...@al...> - 2009-11-24 17:51:16
|
Of course I am far less of an expert at it than you are, but to me, it seems problematic every time I deal with anything other than the standard lisp character set. While the encoding trick applies to input and output streams (sort of), it does not permit all lisp operations to work on all encodings. Also, it requires different compilation (e.g., for unicode), which is to say, I cannot adapt it to the circumstance I encounter when dealing with different input sources. For example, if I want to operate in (UNSIGNED-BYTE 8) throughout my lisp program, there is no way to do it. I cannot do a "(setf " using an (unsigned-byte 8 name) - or a sixbit name - or whatever other coding I may choose, I cannot have a list in which the elements are (unsigned-byte 8) things and have them properly handled by all of the lisp operations, etc. I guess what I am trying to say is that for much of my work (which is largely focussed on dealing with raw datagrams and forensic images), I want to live in a world where all of the byte values can be present in all operations performed. If I input a sequence of bytes, I want to be able to treat them as if they are strings from a standpoint of doing searches, applying regular expressions, etc., and if I want to use then in arrays, all of the operations should operate on them in the same way as if they were regular things (e.g., strings) places in arrays (except of course that they are byte values not restricted to those of strings). Sort, cons, eval, hashes, sequence operations, and everything else should operate exactly as it does with the normal character set, and I should not have to do special escape sequences in order to represent things like a " character or an "ESC" (byte value 127?). While it is relatively easy to input a sequence of bytes into a lisp structure (e.g., an array), once it's in there, it has to be treated specially in every way for every operation, and I always seem to risk a translation that will turn my bytes into some other bytes, which I cannot tolerate for my applications. So, I am sort of looking for a generalization of the original lisp character set - to allow any desired character set to be used for all purposes. Now I realize that this is an unrealistic thing to ask for, and that it won't happen any time soon, but then I also want a built-in GUI that works across all platforms (like java only better), to be able to use unlimited precision arithmetic directly on byte sequences without any special translations, the ability to directly place objects into files (i.e., open a file, write a series of objects into it, and be able to read them all back - random access), the ability to use a file or set of files as cache for memory structures (so that as my programs grow to enormous sizes, lisp automatically does paging to allow me to get to huge calculations and storage sizes without having to do special file IO), a direct database interface for lisp-operated database functions (rather than having to go to MySQL or some such thing), a built-in command interpreter (bash equivalent but operating in lisp), emacs running lisp rather than lisp running in emacs, and of course, world peace. So far, I just keep on writing C interfaces and making everything locally customized, but I would far prefer to live in a lisp environment all the time. I hope this answers your question with regard to the limits on the use of bytes in lisp, and that my response isn't taken in an unfriendly spirit. I recognize the fantastic amount of effort already undertaken and the high quality of clisp as it exists. IT cannot be all things to all people, and I am likely the rare exception rather than the rule with respect to all of these things. But since you asked what I want, I figured I would tell you. Clearly I don't expect to get it. FC On Nov 24, 2009, at 8:51 AM, Sam Steingold wrote: > Fred Cohen wrote: >> Since binary I/O is coming up, I have long wanted to be able to do >> all of the operations I do on ASCII characters and sequences on >> bytes... Somehow it would be nice if we could identify a character >> set (e.g., ASCII / EBCDIC / BYTE / UNICODE / etc.) and have >> everything work in that set instead of in what is something like >> ASCII-7 today. > > I am confused. We already have encodings. > http://clisp.cons.org/impnotes/encoding.html > Is that not what you want? - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Sam S. <sd...@gn...> - 2009-11-25 21:15:49
|
Fred, I am somewhat confused about what your problem is. If you could be more specific (e.g., a code sample), maybe I could be of more help to you here. OTOH, if you are just letting off some frustration, this is fine too. :-) Fred Cohen wrote: > While the encoding trick applies to input and output streams (sort > of), it does not permit all lisp operations to work on all encodings. an encoding is a 1:1 map between a subset of character sequences and a subset of byte sequences (it is also a type comprising all characters s.t. their singletons are in the domain). as such, all relevant lisp operations do work on all encodings. e.g., CONVERT-STRING-FROM-BYTES & CONVERT-STRING-TO-BYTES http://clisp.cons.org/impnotes/encoding.html#string-byte > Also, it requires different compilation (e.g., for unicode), which is > to say, I cannot adapt it to the circumstance I encounter when dealing > with different input sources. without unicode, all characters are 1-byte, so encodings are not particularly useful (unless you are interested in ASCII-incompatible monsters, in which case you are welcome to submit patches to implement these encodings). > For example, if I want to operate in (UNSIGNED-BYTE 8) throughout my > lisp program, there is no way to do it. I cannot do a "(setf " using > an (unsigned-byte 8 name) - or a sixbit name - or whatever other > coding I may choose, I cannot have a list in which the elements are > (unsigned-byte 8) things and have them properly handled by all of the > lisp operations, etc. I am utterly confused here. can you give an example of what you want to do? > I guess what I am trying to say is that for much of my work (which is > largely focussed on dealing with raw datagrams and forensic images), I > want to live in a world where all of the byte values can be present in > all operations performed. If I input a sequence of bytes, I want to be > able to treat them as if they are strings from a standpoint of doing > searches, applying regular expressions, etc., and if I want to use > then in arrays, all of the operations should operate on them in the > same way as if they were regular things (e.g., strings) places in > arrays (except of course that they are byte values not restricted to > those of strings). Sort, cons, eval, hashes, sequence operations, and > everything else should operate exactly as it does with the normal > character set, and I should not have to do special escape sequences in > order to represent things like a " character or an "ESC" (byte value > 127?). so why not read the utf-8 files into character strings? > While it is relatively easy to input a sequence of bytes into a lisp > structure (e.g., an array), once it's in there, it has to be treated > specially in every way for every operation, and I always seem to risk > a translation that will turn my bytes into some other bytes, which I > cannot tolerate for my applications. if you decide on a specific encoding, this should never happen. if you take your chars, convert then to bytes using one encoding, then convert them back to chars using a different encoding, then, yes, you are screwed. > So, I am sort of looking for a generalization of the original lisp > character set - to allow any desired character set to be used for all > purposes. clisp character set includes all the unicode 3.2. do you need some characters which are not there? |
From: <don...@is...> - 2009-11-27 20:56:39
|
- I didn't think that declare optimize had any effect in clisp. http://clisp.podval.org/impnotes/declarations.html lists a very few effects: (OPTIMIZE (SAFETY 3)) function calls are never eliminated SPACE >= 2 documentation string is discarded SPACE >= 3 the original lambda list is also discarded If there are any others I think they should be listed there. - I concluded long ago that what I normally want just can't be done: read every 8bit byte as a distinct character. The problem is not the charsets and encodings but newline handling. The best solution I have found is to read bytes and store their code-char's into the strings your program uses. - For things like http or mail servers it's generally better to use strings internally than byte vectors. For general network traffic monitoring I like bytes better. Although I don't know enough about your specific needs to be sure, I generally think the approach being discussed of implementing a lot of stuff in a new package is going to be counterproductive. Append differs from concatenate in that the last list argument is shared with the result, i.e., not copied. [13]> (string>= o o) 256 CLHS: string>= string1 string2 &key start1 end1 start2 end2 => mismatch-index |
From: Pascal J. B. <pj...@in...> - 2009-11-27 22:11:41
|
On Nov 27, 2009, at 9:00 PM, Don Cohen wrote: > > - I didn't think that declare optimize had any effect in clisp. > http://clisp.podval.org/impnotes/declarations.html > lists a very few effects: > (OPTIMIZE (SAFETY 3)) function calls are never eliminated > SPACE >= 2 documentation string is discarded > SPACE >= 3 the original lambda list is also discarded > If there are any others I think they should be listed there. Right, but we may still use it, for when the sources are compiled with another compiler (or if a new version of clisp becomes more discriminating). > - I concluded long ago that what I normally want just can't be done: > read every 8bit byte as a distinct character. > The problem is not the charsets and encodings but newline handling. > The best solution I have found is to read bytes and store their > code-char's into the strings your program uses. I don't see what the problem with newlines can be. You read bytes, so you have to deal with Carriage Return and Line Feed CODES yourself. > - For things like http or mail servers it's generally better to use > strings internally than byte vectors. For general network traffic > monitoring I like bytes better. It's actually really easy to write the needed reader macros so that in the sources you can use characters, but the program actually deals with ASCII CODES. Notice also that HTTP is NOT a text protocol, even if it uses a lot of ASCII codes. > Although I don't know enough about your specific needs to be sure, > I generally think the approach being discussed of implementing a lot > of stuff in a new package is going to be counterproductive. Well Fred seems to want to do the right thing once for all. I'd tend to agree that there's no need to reimplement every string function, just what is needed by the current program would be enough. -- __Pascal Bourguignon__ http://www.informatimago.com |
From: <don...@is...> - 2009-11-27 22:22:47
|
Pascal J. Bourguignon writes: > > - I concluded long ago that what I normally want just can't be done: > > read every 8bit byte as a distinct character. > > The problem is not the charsets and encodings but newline handling. > > The best solution I have found is to read bytes and store their > > code-char's into the strings your program uses. > > I don't see what the problem with newlines can be. You read bytes, so > you have to deal with Carriage Return and Line Feed CODES yourself. There seems to be no way to read characters and get #\return #\linefeed as a result. If you want strings containing those you have to read bytes and convert them to chars via code-char. > > - For things like http or mail servers it's generally better to use > > strings internally than byte vectors. For general network traffic > > monitoring I like bytes better. > It's actually really easy to write the needed reader macros so that in > the sources you can use characters, but the program actually deals I think this might be a more reasonable approach. > with ASCII CODES. Notice also that HTTP is NOT a text protocol, even > if it uses a lot of ASCII codes. I think http is a text protocol, especially if you view return and linefeed as characters. For instance, the rfc says things like http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]] and Method = "OPTIONS" ; Section 9.2 | "GET" ; Section 9.3 | "HEAD" ; Section 9.4 |
From: Pascal J. B. <pj...@in...> - 2009-11-27 23:30:15
|
On Nov 27, 2009, at 11:22 PM, Don Cohen wrote: > Pascal J. Bourguignon writes: >> with ASCII CODES. Notice also that HTTP is NOT a text protocol, even >> if it uses a lot of ASCII codes. > I think http is a text protocol, especially if you view return and > linefeed as characters. > For instance, the rfc says things like > http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]] > and > Method = "OPTIONS" ; Section 9.2 > | "GET" ; Section 9.3 > | "HEAD" ; Section 9.4 Yes, I know. It's misleading. But if you read closely the RFC, you will see that it's actually a binary protocol. Notably, you can transfer plain binary data without any difficulty. That is not the case for example with SMTP, where the base protocol is textual, and where you have to encode (base64, quoted-printable, whatever) the payload. (email can be sent thru non 8-bit clean channels outside of the Internet). -- __Pascal Bourguignon__ http://www.informatimago.com |
From: Fred C. <fc...@al...> - 2009-11-27 23:49:02
|
On Nov 27, 2009, at 3:30 PM, Pascal J. Bourguignon wrote: > On Nov 27, 2009, at 11:22 PM, Don Cohen wrote: > Pascal J. Bourguignon writes: >>> with ASCII CODES. Notice also that HTTP is NOT a text protocol, >>> even >>> if it uses a lot of ASCII codes. >> I think http is a text protocol, especially if you view return and >> linefeed as characters. >> For instance, the rfc says things like >> http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]] >> and >> Method = "OPTIONS" ; Section 9.2 >> | "GET" ; Section 9.3 >> | "HEAD" ; Section 9.4 > > Yes, I know. It's misleading. But if you read closely the RFC, you > will see that it's actually a binary protocol. Notably, you can > transfer plain binary data without any difficulty. > > That is not the case for example with SMTP, where the base protocol is > textual, and where you have to encode (base64, quoted-printable, > whatever) the payload. (email can be sent thru non 8-bit clean > channels outside of the Internet). So my world may be a bit different from yours. I assume that the bad person on the other side of the connection is not going to follow the rules. As a result, when I read in whatever they send me, regardless of what is supposed to show up, I know that what actually shows up is limited to all byte sequences. So what I don;t want is for my lisp program to fail - especially in a way that allows the bad guy to get into a break or some such thing and allows them to do things that get interpreted - or in a way that stops my program from executing normally. Hence I ALWAYS want bytes, but I want to be able to treat them reasonably using my syntax checks, search for strings, etc. But when I don't find what I am looking for, I also need to be able to handle it properly - hence I want bytes converted to strings so I can use the string functions - but also have the bytes themselves available at all times. FC - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Pascal J. B. <pj...@in...> - 2009-11-28 00:43:59
|
On Nov 28, 2009, at 12:48 AM, Fred Cohen wrote: > On Nov 27, 2009, at 3:30 PM, Pascal J. Bourguignon wrote: >> On Nov 27, 2009, at 11:22 PM, Don Cohen wrote: >> Pascal J. Bourguignon writes: >>>> with ASCII CODES. Notice also that HTTP is NOT a text protocol, >>>> even >>>> if it uses a lot of ASCII codes. >>> I think http is a text protocol, especially if you view return and >>> linefeed as characters. >>> For instance, the rfc says things like >>> http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]] >>> and >>> Method = "OPTIONS" ; Section 9.2 >>> | "GET" ; Section 9.3 >>> | "HEAD" ; Section 9.4 >> >> Yes, I know. It's misleading. But if you read closely the RFC, you >> will see that it's actually a binary protocol. Notably, you can >> transfer plain binary data without any difficulty. >> >> That is not the case for example with SMTP, where the base protocol >> is >> textual, and where you have to encode (base64, quoted-printable, >> whatever) the payload. (email can be sent thru non 8-bit clean >> channels outside of the Internet). > > > So my world may be a bit different from yours. I assume that the bad > person on the other side of the connection is not going to follow > the rules. As a result, when I read in whatever they send me, > regardless of what is supposed to show up, I know that what actually > shows up is limited to all byte sequences. So what I don;t want is > for my lisp program to fail - especially in a way that allows the > bad guy to get into a break or some such thing and allows them to do > things that get interpreted - or in a way that stops my program from > executing normally. Hence I ALWAYS want bytes, but I want to be able > to treat them reasonably using my syntax checks, search for strings, > etc. But when I don't find what I am looking for, I also need to be > able to handle it properly - hence I want bytes converted to strings > so I can use the string functions - but also have the bytes > themselves available at all times. You're perfectly right, you can do that by layering your processing. You can have a presentation layer that will convert the correct bytes to characters, and an application layer that works with lisp characters and strings. However if you want the same performance as C programs, take into account the fact that lisp characters and lisp strings are much more sophisticated than byte vectors, and incur quite an overhead, and at least bigger constants, if not more complexity. Unicode characters are very complex beast, and the complexity is increased when you put them in strings. -- __Pascal Bourguignon__ http://www.informatimago.com |
From: Fred C. <fc...@al...> - 2009-11-28 00:50:50
|
On Nov 27, 2009, at 4:22 PM, Pascal J. Bourguignon wrote: >> ... >> So my world may be a bit different from yours. I assume that the >> bad person on the other side of the connection is not going to >> follow the rules. As a result, when I read in whatever they send >> me, regardless of what is supposed to show up, I know that what >> actually shows up is limited to all byte sequences. So what I don;t >> want is for my lisp program to fail - especially in a way that >> allows the bad guy to get into a break or some such thing and >> allows them to do things that get interpreted - or in a way that >> stops my program from executing normally. Hence I ALWAYS want >> bytes, but I want to be able to treat them reasonably using my >> syntax checks, search for strings, etc. But when I don't find what >> I am looking for, I also need to be able to handle it properly - >> hence I want bytes converted to strings so I can use the string >> functions - but also have the bytes themselves available at all >> times. > > You're perfectly right, you can do that by layering your processing. > > You can have a presentation layer that will convert the correct > bytes to characters, > and an application layer that works with lisp characters and strings. > > However if you want the same performance as C programs, take into > account the fact that lisp characters and lisp strings are much more > sophisticated than byte vectors, and incur quite an overhead, and at > least bigger constants, if not more complexity. > > Unicode characters are very complex beast, and the complexity is > increased when you put them in strings. Ahah! I was unaware of the level of sophistication associated with strings. Perhaps you could point me to a place where I could become enlightened. I don't actually want byte vectors in the sense that C provides them (although many of the benefits like compact storage, type casting, etc. would be very nice to maintain). For example, I would like to be able to efficiently append things without having to define a new larger byte vector. A sort of linked list of byte vectors forming the overall vector would make things far more efficient. A substring could be formed by pointing into the middle of the linked list and having a "length" that defines its extent. A substitution of something of one length with something of a different length could be done by changing the extent of the first part, adding the new insert, and pointing into the last part. You get the idea. I just want the things contained within the strings to be bytes (values 0-255) and not altered by any processing except the specific calls to alter them (no hidden automatic changes please). On the other hand, I hate waste, especially when time and space are tight. As it turns out, clisp versions of lots of things I do are far faster than other language implementations (and far easier to write). But then I run out of space - handling only 200,000 messages, my n^2 algorithms had to be restricted to not perform everything I wanted it to do. If I had the ability to use disk as a cache for lisp RAM content, I could go to far large problem sizes with the full set of functionality - or if I could use the file storage to store structures transparently, I could expand to very large sizes without running out of RAM. As the problem sizes get larger and larger, it would be nice if the algorithms and languages didn't have to change. So my problems reside at both ends of the programming space. At the end with the bits having to be right, language features that make things very convenient for 99.99% of cases, make them fail for the remaining 0.01% - which I ultimately encounter before long. At the end of the sizes getting large, language features that make things very convenient for cases of a few hundred thousand or a million things, make them fail for the things I ultimately encounter at larger sizes and scales before long. Clisp has bignums built in - which is very convenient for many of the large scale things - while C has bytes which are very useful at getting every bit right - java has a GUI that is portable without having to do installs and all sorts of other things at every system I encounter - and the list goes on. What I really want is for lisp to win this bloody thing by getting a little bit better at the bottom end and the top end - so I don't have to keep trying to interface between C for the details, java for the GUI, eternal specialization for large scales, and lisp for expressiveness and efficiency. I guess I haven't yet mentioned the parallel processing thing... but we'll get there... FC > -- > __Pascal Bourguignon__ > http://www.informatimago.com - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Pascal J. B. <pj...@in...> - 2009-11-28 12:10:50
|
On Nov 28, 2009, at 1:50 AM, Fred Cohen wrote: > On Nov 27, 2009, at 4:22 PM, Pascal J. Bourguignon wrote: >>> ... >>> So my world may be a bit different from yours. I assume that the >>> bad person on the other side of the connection is not going to >>> follow the rules. As a result, when I read in whatever they send >>> me, regardless of what is supposed to show up, I know that what >>> actually shows up is limited to all byte sequences. So what I >>> don;t want is for my lisp program to fail - especially in a way >>> that allows the bad guy to get into a break or some such thing and >>> allows them to do things that get interpreted - or in a way that >>> stops my program from executing normally. Hence I ALWAYS want >>> bytes, but I want to be able to treat them reasonably using my >>> syntax checks, search for strings, etc. But when I don't find what >>> I am looking for, I also need to be able to handle it properly - >>> hence I want bytes converted to strings so I can use the string >>> functions - but also have the bytes themselves available at all >>> times. >> >> You're perfectly right, you can do that by layering your processing. >> >> You can have a presentation layer that will convert the correct >> bytes to characters, >> and an application layer that works with lisp characters and strings. >> >> However if you want the same performance as C programs, take into >> account the fact that lisp characters and lisp strings are much >> more sophisticated than byte vectors, and incur quite an overhead, >> and at least bigger constants, if not more complexity. >> >> Unicode characters are very complex beast, and the complexity is >> increased when you put them in strings. > > Ahah! I was unaware of the level of sophistication associated with > strings. Perhaps you could point me to a place where I could become > enlightened. Search about unicode combining characters for example. http://en.wikipedia.org/wiki/Combining_character A single character may need several unicode code points to be represented. So already, you cannot use a scalar to represent unicode characters in general. Then if you want to store the code points in a vector, that means that the index of the vector doesn't necessarily match the index in the string, so string indexing may not be O(1). Of course, the problem is even more complex when you try to fit the unicode model into the lisp model of characters and strings. > [...] I just want the things contained within the strings to be > bytes (values 0-255) and not altered by any processing except the > specific calls to alter them (no hidden automatic changes please). You need to implement your own type to do that, don't use lisp character and string, they're more sophisticated. As mentionned, you can still have "strings" in source files compiled to your own type of strings, with reader macros. > On the other hand, I hate waste, especially when time and space are > tight. As it turns out, clisp versions of lots of things I do are > far faster than other language implementations (and far easier to > write). But then I run out of space - handling only 200,000 > messages, my n^2 algorithms had to be restricted to not perform > everything I wanted it to do. If I had the ability to use disk as a > cache for lisp RAM content, I could go to far large problem sizes > with the full set of functionality - or if I could use the file > storage to store structures transparently, I could expand to very > large sizes without running out of RAM. As the problem sizes get > larger and larger, it would be nice if the algorithms and languages > didn't have to change. > > So my problems reside at both ends of the programming space. At the > end with the bits having to be right, language features that make > things very convenient for 99.99% of cases, make them fail for the > remaining 0.01% - which I ultimately encounter before long. At the > end of the sizes getting large, language features that make things > very convenient for cases of a few hundred thousand or a million > things, make them fail for the things I ultimately encounter at > larger sizes and scales before long. Clisp has bignums built in - > which is very convenient for many of the large scale things - while > C has bytes which are very useful at getting every bit right - java > has a GUI that is portable without having to do installs and all > sorts of other things at every system I encounter - and the list > goes on. > > What I really want is for lisp to win this bloody thing by getting a > little bit better at the bottom end and the top end - so I don't > have to keep trying to interface between C for the details, java for > the GUI, eternal specialization for large scales, and lisp for > expressiveness and efficiency. I guess I haven't yet mentioned the > parallel processing thing... but we'll get there... Are you aware that there are several implementation of Common Lisp, and that clisp is just one implementation of lisp, written in C (hence the name, c lisp)? The point here is that, on 32-bit platforms, clisp is rather more limited in memory than most other implementations. Check array-total- size-limit for example. (On 64-bit platforms, there's enough addressing space to use all the available RAM). If you need to go to the limit of your addressing space, then sbcl or ecl might be better choices than clisp. Also, since you seem to have needs for performance, while this is more difficult to evaluate, it is possible that ecl or sbcl who compile to native code (ecl going thru gcc), could produce faster code than clisp (on the other hand, clisp mustn't be discarted a-priori, since it has now a JIT compiler, and even with a VM, performance can be good for some things. -- __Pascal Bourguignon__ http://www.informatimago.com |
From: Fred C. <fc...@al...> - 2009-11-28 14:35:04
|
On Nov 28, 2009, at 4:09 AM, Pascal J. Bourguignon wrote: > ... > Search about unicode combining characters for example. > http://en.wikipedia.org/wiki/Combining_character > > A single character may need several unicode code points to be > represented. So already, you cannot use a scalar to represent unicode > characters in general. Then if you want to store the code points in a > vector, that means that the index of the vector doesn't necessarily > match the index in the string, so string indexing may not be O(1). > > Of course, the problem is even more complex when you try to fit the > unicode model into the lisp model of characters and strings. Right - but this has nothing to do with the complexity of characters in lisp. There are a virtually unlimited number of different ways to encode information. >> [...] I just want the things contained within the strings to be >> bytes (values 0-255) and not altered by any processing except the >> specific calls to alter them (no hidden automatic changes please). > > You need to implement your own type to do that, don't use lisp > character and string, they're more sophisticated. As mentionned, you > can still have "strings" in source files compiled to your own type of > strings, with reader macros. That's what I was asking about - where can I read about the sophistication of strings in lisp? ... > > Are you aware that there are several implementation of Common Lisp, > and that clisp is just one implementation of lisp, written in C (hence > the name, c lisp)? I am - but I want one that has all of the features of portability, an outstanding team of people behind it, with a long history, open source, the ability to interface to C if need be, willingness to talk to (email with) people like me, in most standard linux distributions, etc. > The point here is that, on 32-bit platforms, clisp is rather more > limited in memory than most other implementations. Check array-total- > size-limit for example. (On 64-bit platforms, there's enough > addressing space to use all the available RAM). All the RAM will not be enough for many of the things I ultimately need to do. When you work with multi-terabyte disks, and you are trying to do analysis linking things across the entire disk to other such things, you need multiple terabytes of memory - which I obviously need to do with file systems. I just want a way to not have to write everything for file systems, RAM, databases, etc. just to get better performance at smaller sizes. Since everybody presumably will run into these limits on their programs at some point, I was thinking that a virtual RAM that extends to disk storage (and ultimately multiple disks across infrastructure) built into lisp would be easier (for me) that doing it myself... and for everyone who ultimately will have to do it themselves as their problems grow in size. > If you need to go to the limit of your addressing space, then sbcl or > ecl might be better choices than clisp. > > Also, since you seem to have needs for performance, while this is more > difficult to evaluate, it is possible that ecl or sbcl who compile to > native code (ecl going thru gcc), could produce faster code than clisp > (on the other hand, clisp mustn't be discarted a-priori, since it has > now a JIT compiler, and even with a VM, performance can be good for > some things. Right - the JIT compiler is extremely helpful since I don't know in advance what my program may have to do. I also do a substantial number of things with eval by building up expressions as I go. That's one of the reasons I prefer an interpretive language - but I still want the performance of compiled code - which is why clisp fits so well for so many things. FC - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Pascal J. B. <pj...@in...> - 2009-11-28 18:42:21
|
On Nov 28, 2009, at 3:34 PM, Fred Cohen wrote: > On Nov 28, 2009, at 4:09 AM, Pascal J. Bourguignon wrote: > >> ... >> Search about unicode combining characters for example. >> http://en.wikipedia.org/wiki/Combining_character >> >> A single character may need several unicode code points to be >> represented. So already, you cannot use a scalar to represent >> unicode >> characters in general. Then if you want to store the code points >> in a >> vector, that means that the index of the vector doesn't necessarily >> match the index in the string, so string indexing may not be O(1). >> >> Of course, the problem is even more complex when you try to fit the >> unicode model into the lisp model of characters and strings. > > Right - but this has nothing to do with the complexity of characters > in lisp. There are a virtually unlimited number of different ways to > encode information. > >>> [...] I just want the things contained within the strings to be >>> bytes (values 0-255) and not altered by any processing except the >>> specific calls to alter them (no hidden automatic changes please). >> >> You need to implement your own type to do that, don't use lisp >> character and string, they're more sophisticated. As mentionned, you >> can still have "strings" in source files compiled to your own type of >> strings, with reader macros. > > That's what I was asking about - where can I read about the > sophistication of strings in lisp? > > ... In the source code of the implementations! Or if you don't wnat to dig there, just compare CHAR_MAX (in C) vs. CHAR-CODE-LIMIT (in CL) and infer the consequences. Compare: int i=42; char c=i; vs. (let ((i 42) (c (make-array 1 :element-type 'character :initial-element #\a))) (setf (aref c 0) i)) and think about what it involves. Hint: to begin with, character in CL will take 32 bits while char in C takes 8 bits, therefore any string manipulation will already four time slower than in C. Of course when doing I/O it also means converting the internal unicode codepoints to the external encoding. In C, if you input utf-8 bytes, you keep them like this (and of course you don't try to take the nth character of a string, since that'd be O(n)). In Lisp, there will be decoding and encoding on each I/O. > All the RAM will not be enough for many of the things I ultimately > need to do. When you work with multi-terabyte disks, and you are > trying to do analysis linking things across the entire disk to other > such things, you need multiple terabytes of memory - which I > obviously need to do with file systems. I just want a way to not > have to write everything for file systems, RAM, databases, etc. just > to get better performance at smaller sizes. Since everybody > presumably will run into these limits on their programs at some > point, I was thinking that a virtual RAM that extends to disk > storage (and ultimately multiple disks across infrastructure) built > into lisp would be easier (for me) that doing it myself... and for > everyone who ultimately will have to do it themselves as their > problems grow in size. Yes, you will probably be disk bound anyway, so clisp will be as good as another implementation. -- __Pascal Bourguignon__ http://www.informatimago.com |
From: Sam S. <sd...@gn...> - 2009-12-02 19:53:40
|
Don Cohen wrote: > Then you need to provide access to other low level facilities that are > not as capable as raw sockets and therefore require less privilege. what system calls are missing? > For instance, you could provide a udp socket interface and there might > be some demand for it among non-root users. I don't even know what > other higher level facilities are available to non-root users. could you please be more specific? what is missing? > How does os:resolve-host-ipaddr find the dns server? it calls gethostbyname or gethostbyaddr. http://clisp.cons.org/impnotes/syscalls.html#resolve-host > Let's see if this helps to clarify the problem. > For every packet you want to send you have to decide on an interface > (device) that will send it. You also have to supply the ethernet > header. This contains a from and to MAC address and two more bytes > which for ip packets will be 0800. > I don't know how to do any of this on non-linux machines. (RAWSOCK:IF-NAME-INDEX) (RAWSOCK:IFADDRS) http://clisp.cons.org/impnotes/rawsock.html#rawsock-high-level > On linux you can normally get a list of the available devices with > ifconfig. I would prefer to use the ip command (see below) but my > impression is that it is not as universally available as ifconfig. > > As a default, at least for addresses out in the internet, > I'd suggest using (if available) > ip route |grep default > which would tell you the default device > in combination with > ip addr > which would tell you its ip and mac address and > ip nei > which would tell you the destination mac address to use. > There are probably equivalent ways of getting that info from /proc but > I don't think those are any more universal. universal or not, parsing the output of a shell command is not what the rawsock module should do. do you need an interface to netlink & netdevice? http://linux.about.com/library/cmd/blcmdl7_netdevice.htm http://linux.about.com/library/cmd/blcmdl7_netlink.htm > Another approach to filling in the same info would be to read packets > coming in from the internet, perhaps in response to a command that you > execute. For instance, if you do an os:resolve-host-ipaddr you can > expect to see a dns reply which will tell you among other things > - the ip address of the dns server > - the interface used to reach it > - the mac addresses you'd need to send packets out that interface > - your ip address (the one to which the reply was sent) this appear to involve catching the packets (frames) that gethostbyname(2) generates, which, I guess, requires running as root. |
From: <don...@is...> - 2009-12-02 21:44:24
|
Sam Steingold writes: Oh, good - this is the message I should have replied to before. So I'll copy the relevant parts of the other message with subject: "[clisp-list] more on raw sockets" to here. > Don Cohen wrote: > > For instance, you could provide a udp socket interface and there might > > be some demand for it among non-root users. I don't even know what > > other higher level facilities are available to non-root users. > > could you please be more specific? > what is missing? I now suspect that you do have all you need: It now occurs to me that you have implemented much more of the socket interface than what I would have called "raw" sockets. It probably is possible to use the non-raw parts to write a version of ping that does not require root. Perhaps this module should be renamed to something more general than "raw". > > How does os:resolve-host-ipaddr find the dns server? > it calls gethostbyname or gethostbyaddr. > http://clisp.cons.org/impnotes/syscalls.html#resolve-host Ok, just another system call that does all of the communication without us ever having to figure out how. > > Let's see if this helps to clarify the problem. > > For every packet you want to send you have to decide on an interface > > (device) that will send it. You also have to supply the ethernet > > header. This contains a from and to MAC address and two more bytes > > which for ip packets will be 0800. > > I don't know how to do any of this on non-linux machines. > (RAWSOCK:IF-NAME-INDEX) > (RAWSOCK:IFADDRS) > http://clisp.cons.org/impnotes/rawsock.html#rawsock-high-level Well, that's interesting, but I'm having a hard time figuring out what the results mean. For me the relevant one seems to be #<RAWSOCK:IFADDRS :NAME "wlan0" :FLAGS (:UP :BROADCAST :RUNNING :MULTICAST 65536) :ADDRESS #<RAWSOCK:SOCKADDR :%DATA #(2 0 0 0 10 0 2 100 0 0 0 0 0 0 0 0)> I gather the first 2 0 corresponds to (2 :INET) The 10 0 2 100 is the ip address of this interface I wonder why it happens to be in the position above. I'd have expected either at the left or right end. :NETMASK #<RAWSOCK:SOCKADDR :%DATA #(2 0 0 0 255 255 255 0 0 0 0 0 0 0 0 0)> :DESTINATION #<RAWSOCK:SOCKADDR :%DATA #(2 0 0 0 10 0 2 255 0 0 0 0 0 0 0 0)> This is the corresponding broadcast address. :DATA NIL> Nothing above tells us the mac address or any reason to choose this interface rather than any of the others. However I now notice that the data does seem related to a different question I asked: When I use (rawsock:socket :packet :raw #x300) instead I get back similar data in the buffer but some other strange data in the address object, such as #<RAWSOCK:SOCKADDR :%DATA #(17 0 8 0 3 0 0 0 33 3 4 6 0 33 107 64)> In this case the data above seem to correspond to #<RAWSOCK:IFADDRS :NAME "wmaster0" :FLAGS (:UP :RUNNING 65536) :ADDRESS #<RAWSOCK:SOCKADDR :%DATA #(17 0 0 0 3 0 0 0 33 3 0 6 0 33 107 64)> :NETMASK NIL :DESTINATION NIL :DATA #<FOREIGN-POINTER #x00007F4808015E88>> I don't know enough about wmaster0 to begin to understand the address above - what I know about it is that it does not follow the expected frame format, which was giving me trouble. But if I look at this result of IFADDRS #<RAWSOCK:IFADDRS :NAME "wlan0" :FLAGS (:UP :BROADCAST :RUNNING :MULTICAST 65536) :ADDRESS #<RAWSOCK:SOCKADDR :%DATA #(17 0 0 0 4 0 0 0 1 0 0 6 0 33 107 64)> The data above does correspond to other data I saw in the address object. The mac address of that interface is 00:21:6B:40:06:7E, or in decimal 0 33 107 64 6 126 -- suspiciously similar but not the same as the data above. :NETMASK NIL :DESTINATION #<RAWSOCK:SOCKADDR :%DATA #(17 0 0 0 4 0 0 0 1 0 0 6 255 255 255 255)> :DATA #<FOREIGN-POINTER #x00007F4808015EE4>> And this gives some hint about the another part of the question: (rawsock:socket :inet :packet #+ignore :all #x300) When I call (rawsock:sockaddr-family device) I get back UNIX on packets that clearly came over the network. The results in this case are actually 1 0 (which I gather means UNIX) and then the character codes for "wlan0" followed by nulls. So perhaps the address data for address family unix is the character name of the interface. But it's not at all clear why using the AF :inet to create the socket should give addresses of family UNIX when using AF :packet to create the socket gives addresses of family PACKET. > > On linux you can normally get a list of the available devices with > > ifconfig. I would prefer to use the ip command (see below) but my > > impression is that it is not as universally available as ifconfig. > > > > As a default, at least for addresses out in the internet, > > I'd suggest using (if available) > > ip route |grep default > > which would tell you the default device > > in combination with > > ip addr > > which would tell you its ip and mac address and > > ip nei > > which would tell you the destination mac address to use. > > There are probably equivalent ways of getting that info from /proc but > > I don't think those are any more universal. > universal or not, parsing the output of a shell command is not what > the rawsock module should do. > do you need an interface to netlink & netdevice? > http://linux.about.com/library/cmd/blcmdl7_netdevice.htm > http://linux.about.com/library/cmd/blcmdl7_netlink.htm Why are these better than shell commands? Do we already have ways to use them from lisp? Perhaps the answer is that we can do that from rawsock, but not in order to figure out what data to pass in to rawsock! > > Another approach to filling in the same info would be to read packets > this appear to involve catching the packets (frames) that gethostbyname(2) > generates, which, I guess, requires running as root. Right, but when I wrote that I was imagining that we were using raw sockets rather than the more general c socket interface. |
From: Sam S. <sd...@gn...> - 2009-12-03 15:39:03
|
Fred Cohen wrote: > (IN-PACKAGE "RAWSOCK") > (setf socket (rawsock:socket :inet :RAW 0 #+ignore :all) > device > (rawsock:make-sockaddr :UNSPEC > (make-array 14 :element-type '(unsigned-byte 8))) > buffer > (make-array 1518 :element-type '(unsigned-byte 8) :fill-pointer 1518))) > (configdev socket "eth0" 1 1 ) ; eth0 no arp raw mode on socket > (defun getraw () > (if (not (= -1 socket)) > (setf (fill-pointer buffer) 1518 > len #+ignore (rawsock::recvfr socket) > (rawsock:recvfrom socket buffer device) > (fill-pointer buffer) len))) > (defun print-buffer () (loop for i below len do (format t "~2,'0X > " (aref buffer i)))) > ;; a rudimentary dump of all packets seen. > (loop while true do ((getraw)(print-buffer))) my sniffer.lisp does not call configdev. how does it affect its functioning? I do not want to try because your explanations did not allay my fears that the changes configdev introduces to the system are completely reversed by closing the socket. |
From: <don...@is...> - 2009-12-03 18:46:43
|
Sam Steingold writes: > > (configdev socket "eth0" 1 1 ) ; eth0 no arp raw mode on socket > my sniffer.lisp does not call configdev. > how does it affect its functioning? Perhaps a better way of thinking about this is how adding that call would affect the result. The only difference, I think, between what tcpdump does and what you do, is that tcpdump puts the interface into "promiscuous" mode. This would cause it to capture packets not intended for it, e.g. those sent to other machines on the lan. This is indeed sometimes what a user of tcpdump wants, but not always. If you're only trying to debug code on your own machine (or one of its peers) then you don't really care about all the packets being sent to other machines. In modern networks normal machines tend not to get much traffic sent to other machines anyway (the difference between a switch and a hub). Tcpdump has a parameter for controlling promiscuous mode, -p. Another small point, your code captures packets to ALL interfaces. Normally tcpdump only captures packets to one. In tcpdump you can specify the device "any" but in that case it does not use promiscuous mode. In fact, even Fred's code only did that for one specific device, eth0. If you want to include packets not sent to the interface then you can put it into promiscuous mode by doing something like ifconfig eth0 promisc then run your sniffer, then ifconfig eth0 -promisc The stuff about no arp is not something you would want to do in normal operation. I don't know what else configdev might do. In summary your sniffer is something like tcpdump -c 100 -lenx -s 1518 -i any A little better in that it shows the interface, a little worse in that it has no timestamps and does no protocol decoding. |
From: Fred C. <fc...@al...> - 2009-12-03 16:02:18
|
My advice to you and all others is to test all such things on a system that you can easily reboot and that is not critical to anything before making it part of any operational system or system used for any other purpose. My implementations of this technology tend to be on systems with bootable CDs and floppy-based configuration files (write locked except during down-time). Thus I can push reset and try again on a moment's notice. Improper use may disrupt not only the local machine, but also surrounding networks, and who knows what else. Also - in trying your test script, even after recompiling clisp with rawsock built in, it failed... >sudo clisp -K full /u/fc/lisp/sniffer.lsp inet 100 *** - UNIX error 43 (EPROTONOSUPPORT): Protocol not supported >sudo clisp -K full /u/fc/lisp/sniffer.lsp packet 100 *** - RAWSOCK:SOCKET: Lisp value :PACKET is not found in table "check_socket_domain": ((0 :UNSPEC) (1 :UNIX) (1 :LOCAL) (2 :INET) (3 :IMPLINK) (4 :PUP) (5 :CHAOS) (9 :DATAKIT) (10 :CCITT) (23 :IPX) (6 :NS) (7 :ISO) (7 :OSI) (8 :ECMA) (16 :APPLETALK) (30 :INET6) (12 :DECNET) (13 :DLI) (14 :LAT) (15 :HYLINK) (17 :ROUTE) (11 :SNA) (33 :NETBIOS)) I tried various combinations including altering the values within the program... (defparameter *socket* (let ((arg (pop *args*))) (cond ((string= arg "inet") (rawsock:socket :inet :dgram #x300)) ((string= arg "packet") (rawsock:socket :packet :raw #x300)) (t (error "invalid socket argument ~S" arg))))) (unless (plusp *socket*) (error "Invalid socket created for ~S: ~S" *args* *socket*)) Somehow, a demo should work across all such situations... This seems to not error out on my system: (defparameter *socket* (let ((arg (pop *args*))) (cond ((string= arg "inet") (rawsock:socket :inet :dgram 0 #+ignore #x300)) ((string= arg "packet") (rawsock:socket :inet :raw 0 #+ignore #x300)) (t (error "invalid socket argument ~S" arg))))) (unless (plusp *socket*) (error "Invalid socket created for ~S: ~S" *args* *socket*)) FC On Dec 3, 2009, at 7:38 AM, Sam Steingold wrote: > Fred Cohen wrote: >> (IN-PACKAGE "RAWSOCK") >> (setf socket (rawsock:socket :inet :RAW 0 #+ignore :all) >> device >> (rawsock:make-sockaddr :UNSPEC >> (make-array 14 :element-type '(unsigned-byte 8))) >> buffer >> (make-array 1518 :element-type '(unsigned-byte 8) :fill-pointer >> 1518))) >> (configdev socket "eth0" 1 1 ) ; eth0 no arp raw mode on socket >> (defun getraw () >> (if (not (= -1 socket)) >> (setf (fill-pointer buffer) 1518 >> len #+ignore (rawsock::recvfr socket) >> (rawsock:recvfrom socket buffer device) >> (fill-pointer buffer) len))) >> (defun print-buffer () (loop for i below len do (format t "~2,'0X >> " (aref buffer i)))) >> ;; a rudimentary dump of all packets seen. >> (loop while true do ((getraw)(print-buffer))) > > my sniffer.lisp does not call configdev. > how does it affect its functioning? > I do not want to try because your explanations did not allay my > fears that the changes configdev introduces to the system are > completely reversed by closing the socket. > > > - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Sam S. <sd...@gn...> - 2009-12-03 17:31:38
|
Fred Cohen wrote: > My advice to you and all others is to test all such things on a system > that you can easily reboot and that is not critical to anything before > making it part of any operational system or system used for any other > purpose. My implementations of this technology tend to be on systems > with bootable CDs and floppy-based configuration files (write locked > except during down-time). Thus I can push reset and try again on a > moment's notice. Improper use may disrupt not only the local machine, > but also surrounding networks, and who knows what else. I put it into the README. still, resetting options should be possible. I will take a look at it. > This seems to not error out on my system: > (defparameter *socket* > (let ((arg (pop *args*))) > (cond ((string= arg "inet") > (rawsock:socket :inet :dgram 0 #+ignore #x300)) never prints anything > ((string= arg "packet") > (rawsock:socket :inet :raw 0 #+ignore #x300)) (EPROTONOSUPPORT): Protocol not supported with protocol #x300: never prints anything > (t (error "invalid socket argument ~S" arg))))) > (unless (plusp *socket*) > (error "Invalid socket created for ~S: ~S" *args* *socket*)) you can do sniffer.lisp -type raw -protocol 0 sniffer.lisp -type dgram -protocol 0 &c. |
From: Fred C. <fc...@al...> - 2009-12-03 17:54:38
|
> > ... >> This seems to not error out on my system: >> (defparameter *socket* >> (let ((arg (pop *args*))) >> (cond ((string= arg "inet") >> (rawsock:socket :inet :dgram 0 #+ignore #x300)) > > never prints anything You have to be patient - it prints ping responses... > >> ((string= arg "packet") >> (rawsock:socket :inet :raw 0 #+ignore #x300)) > > (EPROTONOSUPPORT): Protocol not supported > > with protocol #x300: never prints anything You have to be patient - it prints ping responses... > >> (t (error "invalid socket argument ~S" arg))))) >> (unless (plusp *socket*) >> (error "Invalid socket created for ~S: ~S" *args* *socket*)) > > you can do > > sniffer.lisp -type raw -protocol 0 > sniffer.lisp -type dgram -protocol 0 > > &c. Perhaps it would be a better idea to allow the system to identify the valid values for the parameters (they are system-dependent) and then allow the user to specify what they want. Specifically: sniffer.lisp help might provide the allowable values for each of the arguments and list available interfaces, and then the user could specify desired parameters on the command line. OF course I don;t know how to do this - even though the error messages seem to indicate that lisp knows what the possible answers are... FC - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: Sam S. <sd...@gn...> - 2009-12-03 18:10:00
|
Fred Cohen wrote: >> ... >>> This seems to not error out on my system: >>> (defparameter *socket* >>> (let ((arg (pop *args*))) >>> (cond ((string= arg "inet") >>> (rawsock:socket :inet :dgram 0 #+ignore #x300)) >> never prints anything > > You have to be patient - it prints ping responses... even if I ping the local host (from a different machine) it does not print anything. >>> ((string= arg "packet") >>> (rawsock:socket :inet :raw 0 #+ignore #x300)) >> (EPROTONOSUPPORT): Protocol not supported >> >> with protocol #x300: never prints anything > > You have to be patient - it prints ping responses... ditto. > Perhaps it would be a better idea to allow the system to identify the > valid values for the parameters (they are system-dependent) and then > allow the user to specify what they want. Specifically: sniffer.lisp > help might provide the allowable values for each of the arguments and > list available interfaces, and then the user could specify desired > parameters on the command line. OF course I don;t know how to do this > - even though the error messages seem to indicate that lisp knows what > the possible answers are... yes, and I also can add help, and menu, and completion, however, polishing UI is not the goal here. I want to demonstrate what one can do with rawsock, not to provide a perfect end-user application. so, who is writing the ntp and dns clients? you or Don? |
From: Fred C. <fc...@al...> - 2009-12-03 18:30:45
|
We have long had DNS operating with rawsock - as well as lots of other stuff - but the problem, is local configuration. Because everything is so system- and situation-specific, and because testing these things is so tricky, we have not created a generalized version. I am testing out the new rawsock now with code that worked under the previous (really old) version and it is problematic under OSX - now recompiling clisp on a Linux box to test it there... I wish the standard build had it all included... but then why would we need me? FC On Dec 3, 2009, at 10:09 AM, Sam Steingold wrote: > Fred Cohen wrote: >>> ... >>>> This seems to not error out on my system: >>>> (defparameter *socket* >>>> (let ((arg (pop *args*))) >>>> (cond ((string= arg "inet") >>>> (rawsock:socket :inet :dgram 0 #+ignore #x300)) >>> never prints anything >> You have to be patient - it prints ping responses... > > even if I ping the local host (from a different machine) > it does not print anything. > >>>> ((string= arg "packet") >>>> (rawsock:socket :inet :raw 0 #+ignore #x300)) >>> (EPROTONOSUPPORT): Protocol not supported >>> >>> with protocol #x300: never prints anything >> You have to be patient - it prints ping responses... > > ditto. > >> Perhaps it would be a better idea to allow the system to identify >> the valid values for the parameters (they are system-dependent) >> and then allow the user to specify what they want. Specifically: >> sniffer.lisp help might provide the allowable values for each of >> the arguments and list available interfaces, and then the user >> could specify desired parameters on the command line. OF course I >> don;t know how to do this - even though the error messages seem to >> indicate that lisp knows what the possible answers are... > > yes, and I also can add help, and menu, and completion, however, > polishing UI is not the goal here. > I want to demonstrate what one can do with rawsock, not to provide a > perfect end-user application. > so, who is writing the ntp and dns clients? you or Don? > > > > - This communication is confidential to the parties it is intended to serve - Fred Cohen & Associates tel/fax: 925-454-0171 http://all.net/ 572 Leona Drive Livermore, CA 94550 Join http://tech.groups.yahoo.com/group/FCA-announce/join for our mailing list |
From: <don...@is...> - 2009-12-03 19:50:43
|
Fred Cohen writes: > Also - in trying your test script, even after recompiling clisp with > rawsock built in, it failed... BTW I should report success on my machine with this: $ clisp -K full /tmp/sniffer.lisp inet 10 > >sudo clisp -K full /u/fc/lisp/sniffer.lsp inet 100 > *** - UNIX error 43 (EPROTONOSUPPORT): Protocol not supported > > >sudo clisp -K full /u/fc/lisp/sniffer.lsp packet 100 > *** - RAWSOCK:SOCKET: Lisp value :PACKET is not found in table > "check_socket_domain": > ((0 :UNSPEC) (1 :UNIX) (1 :LOCAL) (2 :INET) (3 :IMPLINK) > (4 :PUP) (5 :CHAOS) (9 :DATAKIT) (10 :CCITT) (23 :IPX) (6 :NS) (7 :ISO) > (7 :OSI) (8 :ECMA) (16 :APPLETALK) (30 :INET6) (12 :DECNET) > (13 :DLI) (14 :LAT) (15 :HYLINK) (17 :ROUTE) (11 :SNA) (33 :NETBIOS)) Wow, that's quite a bit different from what I see: "check_socket_domain": ((0 :UNSPEC) (1 :UNIX) (1 :LOCAL) (2 :INET) (3 :AX25) (4 :IPX) (5 :APPLETALK) (6 :NETROM) (7 :BRIDGE) (8 :ATMPVC) (9 :X25) (10 :INET6) (11 :ROSE) (12 :DECNET) (13 :NETBEUI) (14 :SECURITY) (15 :KEY) (16 :NETLINK) (16 :ROUTE) (17 :PACKET) (18 :ASH) (19 :ECONET) (20 :ATMSVC) (22 :SNA) (23 :IRDA) (24 :PPPOX) (25 :WANPIPE) (31 :BLUETOOTH)) Let me guess - you're using a mac? > Somehow, a demo should work across all such situations... Clearly not this sort of demo. > (rawsock:socket :inet :dgram 0 #+ignore #x300)) My man ip(7) says An IP socket is created by calling the socket(2) function as socket(AF_INET, socket_type, protocol). Valid socket types are SOCK_STREAM to open a tcp(7) socket, SOCK_DGRAM to open a udp(7) socket, or SOCK_RAW to open a raw(7) socket to access the IP protocol directly. protocol is the IP protocol in the IP header to be received or sent. The only valid values for protocol are 0 and IPPROTO_TCP for TCP sockets, and 0 and IPPROTO_UDP for UDP sockets. For SOCK_RAW you may specify a valid IANA IP protocol defined in RFC 1700 assigned num- bers. Maybe it's different on your machine. This seems to mean that your line above should be watching for udp. When I try that I get nothing - no udp, no icmp. When I try (rawsock:socket :inet :raw 0) I get protocol not supported which is strange since (rawsock:socket :inet :raw t) *** - RAWSOCK:SOCKET: Lisp value T is not found in table "check_socket_protocol": ((0 :IPPROTO-IP) (41 :IPPROTO-IPV6) (1 :IPPROTO-ICMP) (255 :IPPROTO-RAW) (6 :IPPROTO-TCP) (17 :IPPROTO-UDP) (2 :IPPROTO-IGMP) (4 :IPPROTO-IPIP) On the other hand rfc 1700 (which I think is now obsolete) does say Decimal Keyword Protocol References ------- ------- -------- ---------- 0 Reserved [JBP] 1 ICMP Internet Control Message [RFC792,JBP] 2 IGMP Internet Group Management [RFC1112,JBP] 3 GGP Gateway-to-Gateway [RFC823,MB] 4 IP IP in IP (encasulation) [JBP] ... So I think the problem is in the (0 :IPPROTO-IP) above. When I use (rawsock:socket :inet :raw 1) I do indeed see icmp and 17 shows udp. |