Search Results for "gnu make"

sperm is a curated repository that gathers a collection of notable articles related to reverse engineering and software analysis. It primarily acts as a knowledge archive where previously published technical posts are compiled and organized for easier access and long-term reference. These articles originate from multiple technical communities and platforms and are exported into Markdown format to maintain a consistent and readable structure. sperm focuses on educational material that...

Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by...

Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization. Make sure that Scrapyd has been installed and started on all of your hosts. Start ScrapydWeb via command scrapydweb. (a config file would be generated for customizing settings on the first startup.) Add your Scrapyd servers, both formats of string and tuple are supported, you can attach basic auth for accessing the Scrapyd server, as well as a string for grouping or labeling. You can select any...

A C# coded web crawler that analyzes pagerank, total links, no follow links, and make a ranking for their best-positioned pages in Google

It is basicly a program that can make you a search engine. It is a web crawler, has all the web site source code (in ASP, soon to be PHP as well), and a mysql database.