OpenWebSpider

File Release Notes and Changelog

Release Name: 0.5.1

Notes:
openwebspider-0.5.1\alloca.c                                   
openwebspider-0.5.1\doc\cache.php.txt                          
openwebspider-0.5.1\doc\Changelog.txt                          
openwebspider-0.5.1\doc\openwebspider-0.5-handbook.htm         
openwebspider-0.5.1\doc\search.php.txt                         
openwebspider-0.5.1\doc\searchfunct.php.txt                    
openwebspider-0.5.1\doc\vars.php.txt                           
openwebspider-0.5.1\functions.h                                
openwebspider-0.5.1\getopt.c                                   
openwebspider-0.5.1\getopt.h                                   
openwebspider-0.5.1\gui.h                                      
openwebspider-0.5.1\hstlist.h                                  
openwebspider-0.5.1\htmlfnct.h                                 
openwebspider-0.5.1\libmysql.lib                               
openwebspider-0.5.1\list.h                                     
openwebspider-0.5.1\macro\macro.dsp                            
openwebspider-0.5.1\macro\macro.dsw                            
openwebspider-0.5.1\macro\ows_macro.c                          
openwebspider-0.5.1\Makefile                                   
openwebspider-0.5.1\misc.h                                     
openwebspider-0.5.1\modules\modHeader.h                        
openwebspider-0.5.1\modules\regexFilter.c                      
openwebspider-0.5.1\modules\regexFilter.def                    
openwebspider-0.5.1\modules\regexFilter.dsp                    
openwebspider-0.5.1\modules\regexFilter.dsw                    
openwebspider-0.5.1\modules\sampleModFilter.c                  
openwebspider-0.5.1\modules.h                                  
openwebspider-0.5.1\mod_regex.conf                             
openwebspider-0.5.1\mymutex.h                                  
openwebspider-0.5.1\openwebspider-0.5.c                        
openwebspider-0.5.1\openwebspider.conf                         
openwebspider-0.5.1\openwebspider.dsp                          
openwebspider-0.5.1\openwebspider.dsw                          
openwebspider-0.5.1\options.h                                  
openwebspider-0.5.1\rank.h                                     
openwebspider-0.5.1\regex.c                                    
openwebspider-0.5.1\regex.h                                    
openwebspider-0.5.1\robots.h                                   
openwebspider-0.5.1\snprintf.c                                 
openwebspider-0.5.1\socket.h                                   
openwebspider-0.5.1\sqlfnct.h                                  
openwebspider-0.5.1\strfnct.h                                  
openwebspider-0.5.1\temptable.h                                
openwebspider-0.5.1\thread.h                                   
openwebspider-0.5.1\urlfunct.h                                 
openwebspider-0.5.1\Win32_bin\                                 
openwebspider-0.5.1\Win32_bin\macro.exe                        
openwebspider-0.5.1\Win32_bin\openwebspider.exe                
openwebspider-0.5.1\Win32_bin\regexFilter.dll                  
openwebspider-0.5.1\xmalloc.c                                  
                                                              

Changes:
OpenWebSpider v0.5.1
- Extended the set of the characters indexed
- Fixed a bug in InitIndexing() and ReturnFirstUrl()
- Optimized KillThreads()
- Optimized the module handler
- Modified modRegexFilter and mod_regex.conf structure

OpenWebSpider v0.5
- New argument: '-f' (Support for external modules (.dll OR .so))
- New argument: '-d' (Crawl Delay in MilliSeconds) [Default: 0 (No Delay)] [Thanks to Craig Atkins for the feedback]
- New argument: '-l' (Limit the maximum number of pages indexed per site) [Default: 0 (No limit)] [Thanks to Constantinos Laitsas for the feedback]
- New argument: '-x' (store full html of pages)
- New argument: '-z' (store full html of pages (compressed))
- New argument: '-u' (index only new pages (Update))
- New argument: '-r' (Saves relationships between page; who links who and who is linked from)
- robots.txt: Support for "Crawl-Delay" [Thanks to Craig Atkins for the feedback]
- Support for tag STYLE in UnHtml() [Thanks to Constantinos Laitsas for the feedback]
- Support for tag BASE (<base href="http://www.openwebspider.org/">) in ParseUrl()
- Fixed a bug in FlushTempTable() [Thanks to Constantinos Laitsas for the feedback]
- Fixed a bug in ParseHTTPRequest() [Thanks to Constantinos Laitsas for the feedback]
- Optimized AddExternalHost()
- Optimized pRelationships()
- Optimized GetTickCount() for linux! now returns milliseconds
- Optimized UnHtml()
- Optimized OnlyOneSpace()
- Optimized RemoveShit()
- Optimized IndexPage()
- Optimized thrdBlock()

OpenWebSpider v0.4.1
- Support for tag IFRAME

OpenWebSpider v0.4
- GUI
- Support for "starting/stopping" threads
- Support for DELAYED INSERT
- Added parameter "-e"; using it the spider doesn't add external host to the DB
- Fixed a memory leak in IndexPage()
- Fixed a memory leak in ReturnFirstUrl()
- Fixed a memory leak in AddExternalHost()
- Fixed a memory leak in GetHostRank()

OpenWebSpider v0.3
- Support for robots.txt
- Fixed a bug in function BetweenTag
- Support for levels   (see related docs)
- Support for HostRank (see related docs)
- Support for PageRank (see related docs)
- Added parameter -m   (see ows_arguments)
- Fixed a bug in IndexPage

OpenWebSpider v0.2.1
- Support for HTTP 302
- Corrected some bugs

OpenWebSpider v0.2
- Multithreaded
- Corrected a lot of bugs
- Most functions rewritten
- List of hosts
- Changed the indexing-mode
- Changed the Search engine (Full-text)
- Removed realtime search