Search Results for "spider"

DotnetSpider is a web crawling and data extraction framework built on the .NET Standard platform. It is designed to help developers create efficient and scalable crawlers for collecting structured data from websites. It provides a high-level API that simplifies the process of defining spiders, managing requests, and extracting content from web pages. Developers can create custom spiders by extending base classes and configuring pipelines that handle downloading, parsing, and storing...

BlackWidow is a Python-based web application scanning tool designed to crawl target websites and collect open-source intelligence (OSINT) while identifying potential security vulnerabilities. It functions as a web spider that systematically explores a site to gather valuable information such as URLs, dynamic parameters, subdomains, email addresses, and phone numbers associated with the target domain. By automatically extracting this data, BlackWidow helps security professionals and researchers build a clearer understanding of a website’s structure and publicly accessible information. ...

NZBGetter is a PHP Script for linux based systems to spider NZB index sites for NZB files matching your predefined search patterns. The script downloads matching NZB files and passes them to your Usenet Reader.

Funnel is a project for use on intranets, or selected sites on the Internet to gather together and index information from several different sources and make it available through a sane, usable interface.