Search Results for "raw"

...Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by avoiding the noise caused by recurring elements (headers, footers, links/blogroll etc.) and second by including information such as author and date in order to make sense of the data. The extractor tries to strike a balance between limiting noise (precision) and including all valid parts (recall). ...

...It is designed for users who run a server component on a VPS and a local client on Windows, macOS, or Linux. The tool supports two different approaches: Paqet, which uses KCP over raw sockets for simpler cases, and GFW-Knocker, which combines TCP and QUIC tunneling for heavier censorship scenarios. It provides guided installation, configuration, service management, status checks, and backup-oriented dual-backend operation. The project focuses on making complex proxy infrastructure easier to install and operate from a single shell-based interface. ...

...These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages. Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. It also contains examples of proxy pool integration and encapsulation to support more reliable crawling when working with sites that enforce request limits.

A HTML scraper that uses machine learning frameworks to extract labelled fields from raw HTML. The project also involves the development of a tool to display the semi structured data generated by the scraper component.

pyip is a Python package offering assembling/disassembling of raw ip packet, including ip, udp, and icmp. Also it includes 2 utilities based on raw ip, traceroute and ping.

zSearch is a simple python based crawler and search engine. Raw HTML are stored in bzip2 archives, the index is created using pylucene, and twsited is used to provide internal http server. Results are sent back as XML over HTTP.