Menu
▾
▴
pyzor-users
|
From: Matt <ma...@gm...> - 2009-08-14 04:41:37
|
Hi During my limited attempts to port the basic check routines over to .NET i noticed that there are no minimum requirements before the hash is calculated. Would this not generate false positives when you have empty messages or message with very little text? Thks |
|
From: Tony M. <to...@sp...> - 2009-08-14 05:03:10
|
> During my limited attempts to port the basic check routines over to .NET Which .NET language are you porting to? > i noticed that there are no minimum requirements before the hash is > calculated. That's not completely correct. There has to be at least one line whose normalised length is 8 characters or more, otherwise there are no offsets, and no digest. A message with very little text will have a completely different digest to a different message with very little text. However, the basic point is correct - the smaller the message, the less unique the hash is. As I've indicated previously, I think the digest specification needs re-examining, but I don't think it's something that I should or will get to this year. Cheers, Tony |
|
From: Matt <ma...@gm...> - 2009-08-14 07:08:02
|
Ok thanks for the tip. Running on C#, VB.NET. And on a side note i have managed to get Pyzor to partially run on windows by uncommenting out certain lines so that it does not throw errors with python26 I have managed to get it down to the basics, the only thing i cannot find an equivalent of how pyzor 'normalizes the html' in .net. I have this regex snippet 'html_tag_ptrn = re.compile(r'<.*?>')' in pyzor but using the same snippet does not produce the desired results. Any idea? So far this is what i have done as best as i can understand 1. Removes(any) 'words' (sequences of characters separated by whitespace) that are 10 or more characters 2. Remove anything that is so long it that it looks like a unique identifier 3. Removes anything that looks like an email address 4. Removes anything that looks like a URL. 5. Rmoves anything that looks like HTML tags. (STUCK HERE!) 6. Removes any whitespace. 7. Discards any lines that are fewer than 8 characters in length. 8. Removes extra lines Then run the following rules : 1. If the message is greater than 4 lines in length, do the following: - Discard the first 20% of the message - then Grab the next 3 lines. - Discards the 60% of the message - then Grab the next 3 lines. - Discards the remainder of the message. If less than 4 lines use the entire body Am i missing anything else? On Fri, Aug 14, 2009 at 1:02 PM, Tony Meyer <to...@sp...> wrote: > > During my limited attempts to port the basic check routines over to .NET > > Which .NET language are you porting to? > > > i noticed that there are no minimum requirements before the hash is > > calculated. > > That's not completely correct. There has to be at least one line > whose normalised length is 8 characters or more, otherwise there are > no offsets, and no digest. A message with very little text will have > a completely different digest to a different message with very little > text. > > However, the basic point is correct - the smaller the message, the > less unique the hash is. As I've indicated previously, I think the > digest specification needs re-examining, but I don't think it's > something that I should or will get to this year. > > Cheers, > Tony > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus > on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > pyzor-users mailing list > pyz...@li... > https://lists.sourceforge.net/lists/listinfo/pyzor-users > |
|
From: Tony M. <to...@sp...> - 2009-08-16 03:15:42
|
> Running on C#, VB.NET. You've got both a C# and a Visual Basic implementation? Couldn't you just have one and then use an assembly from the other (or any .NET) language? > And on a side note i have > managed to get Pyzor to partially run on windows by uncommenting out certain > lines so that it does not throw errors with python26 There are only really two issues with running Pyzor on Windows: it currently uses signal.alarm to handle timeouts, and it assumes POSIX-style paths for various files. The latter is easily fixed (I'll try to get this for 0.6, although I'm already way behind the time I wanted to have that done). The former can be fixed in various ways - e.g. having platform-specific timeout code, or using threads rather than signals, or just not having a timeout on platforms without signal.alarm (leaving handling timeouts to the user). Can I ask what you're planning to do with your implementation when it's done? In particular: are you planning on distributing it? If so, then the best solution might be for the Python pyzor to stay reasonably unfriendly to Windows and just provide links to your implementation. (And ensure that we work with you to make sure that the implementations stay reasonably in sync). > I have managed to get it down to the basics, the only thing i cannot find > an equivalent of how pyzor 'normalizes the html' in .net. I have this regex > snippet 'html_tag_ptrn = re.compile(r'<.*?>')' in pyzor but using the same > snippet does not produce the desired results. Any idea? That regular expression captures anything (other than newlines) within angle brackets (the *? makes it a non-greedy capture, which means it'll stop at the first >, rather than the last), including <> (i.e. nothing between the brackets). Again, this is a very crude expression, that will catch things like <this> as well as real tags. It also completely ignores the MIME type of the message, so this runs on both text/plain and text/html. > So far this is what i have done as best as i can understand > > 1. Removes(any) 'words' (sequences of characters separated by whitespace) > that are 10 or more characters > 2. Remove anything that is so long it that it looks like a unique identifier 1 & 2 are the same thing, really. i.e. 2 is done by doing 1. > 3. Removes anything that looks like an email address Yes. This, like the URL regex, isn't crafted amazingly well. "looks like an email address" just means any non-whitespace characters that surround an "@". I suppose it's good enough and doesn't really effect the uniqueness much, but it's not the regex I would choose. > 4. Removes anything that looks like a URL. Yes. This regex is worse than the email one. When we get to re-examining the specification, I'd like to change this to something more accurate. At the moment, it's any sequence of lower-case letters followed by a colon and then a sequence of non-whitespace characters. > 6. Removes any whitespace. > 7. Discards any lines that are fewer than 8 characters in length. Yes. > 8. Removes extra lines What do you mean by this? Cheers, Tony |
|
From: matply <ma...@gm...> - 2009-08-16 04:19:53
|
Hi 1. When I mentioned C# or VB.NET I meant it more as either language. I am coding in VB.Net primarily 2. Thank you for the other explanations. Makes a lot of sense 3. If there is demand for a .Net equivalent then I do not mind releasing this public domain (the client that is). However as you have pointed I feel that this should probably wait till 0.6. For cross portability reasons I would like to suggest that perhaps the HASH lookups for the Pyzor client could be done in a reverse DNS style (i.e like how it is done in IXHASH). This would make it easier to maintain other client/port that communicates with the Pyzor servers. Having said that though, if Pyzor could run natively in Windows then this would be a better alternative. 4. When I mentioned remove extra lines. I meant remove any extra line break at the top and bottom of the message. Thanks -Matt -----Original Message----- From: Tony Meyer [mailto:to...@sp...] Sent: Sunday, August 16, 2009 11:15 AM To: pyz...@li... Subject: Re: Message Prequalification for Digest > Running on C#, VB.NET. You've got both a C# and a Visual Basic implementation? Couldn't you just have one and then use an assembly from the other (or any .NET) language? > And on a side note i have > managed to get Pyzor to partially run on windows by uncommenting out certain > lines so that it does not throw errors with python26 There are only really two issues with running Pyzor on Windows: it currently uses signal.alarm to handle timeouts, and it assumes POSIX-style paths for various files. The latter is easily fixed (I'll try to get this for 0.6, although I'm already way behind the time I wanted to have that done). The former can be fixed in various ways - e.g. having platform-specific timeout code, or using threads rather than signals, or just not having a timeout on platforms without signal.alarm (leaving handling timeouts to the user). Can I ask what you're planning to do with your implementation when it's done? In particular: are you planning on distributing it? If so, then the best solution might be for the Python pyzor to stay reasonably unfriendly to Windows and just provide links to your implementation. (And ensure that we work with you to make sure that the implementations stay reasonably in sync). > I have managed to get it down to the basics, the only thing i cannot find > an equivalent of how pyzor 'normalizes the html' in .net. I have this regex > snippet 'html_tag_ptrn = re.compile(r'<.*?>')' in pyzor but using the same > snippet does not produce the desired results. Any idea? That regular expression captures anything (other than newlines) within angle brackets (the *? makes it a non-greedy capture, which means it'll stop at the first >, rather than the last), including <> (i.e. nothing between the brackets). Again, this is a very crude expression, that will catch things like <this> as well as real tags. It also completely ignores the MIME type of the message, so this runs on both text/plain and text/html. > So far this is what i have done as best as i can understand > > 1. Removes(any) 'words' (sequences of characters separated by whitespace) > that are 10 or more characters > 2. Remove anything that is so long it that it looks like a unique identifier 1 & 2 are the same thing, really. i.e. 2 is done by doing 1. > 3. Removes anything that looks like an email address Yes. This, like the URL regex, isn't crafted amazingly well. "looks like an email address" just means any non-whitespace characters that surround an "@". I suppose it's good enough and doesn't really effect the uniqueness much, but it's not the regex I would choose. > 4. Removes anything that looks like a URL. Yes. This regex is worse than the email one. When we get to re-examining the specification, I'd like to change this to something more accurate. At the moment, it's any sequence of lower-case letters followed by a colon and then a sequence of non-whitespace characters. > 6. Removes any whitespace. > 7. Discards any lines that are fewer than 8 characters in length. Yes. > 8. Removes extra lines What do you mean by this? Cheers, Tony ---------------------------------------------------------------------------- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ pyzor-users mailing list pyz...@li... https://lists.sourceforge.net/lists/listinfo/pyzor-users |
|
From: matply <ma...@gm...> - 2009-08-15 11:20:03
|
Hi How does Pyzor deal with Non English Spam, for Example Chinese characters or other languages which do not use spaces for their sentences? If all characters more than 10 characters are removed we are most likely left with a very empty body to hash? Thanks |
|
From: Tony M. <to...@sp...> - 2009-08-15 20:15:29
|
> How does Pyzor deal with Non English Spam, for Example Chinese characters or > other languages which do not use spaces for their sentences? If all > characters more than 10 characters are removed we are most likely left with > a very empty body to hash? Pyzor completely ignores the language. Non-English languages that do use whitespace to separate out words will generally work fine, although the average word length in other languages is often longer than in English, so normalisation may remove content that would be better left in the digest. As you suggested, if there is little or no whitespace, as with many Eastern languages, there may be little content to digest. This is something that could be considered when looking into the specification (probably early next year). Until then, you can (a) submit a patch - or at least open a ticket - if this is important to you, and/or (b) adjust the normalisation settings in your copy of Pyzor to better match the messages you are trying to identify (of course, you'll need to have mutliple sources doing this in order to match with them). Cheers, Tony |
Thanks for helping keep SourceForge clean.
X