Search Results for "crawling"
Sort By:
39 projects for "crawling" with 1 filter applied:
-
Context for your AI agentsBuild data pipelines that feed your AI models and agents without managing infrastructure. Crawl any website, transform content, and push directly to your preferred vector store. Use 10,000+ tools for RAG applications, AI assistants, and real-time knowledge bases. Monitor site changes, trigger workflows on new data, and keep your AIs fed with fresh, structured information. Cloud-native, API-first, and free to start until you need to scale.
-
Atera all-in-one platform IT management software with AI agentsAtera’s AI agents don’t just assist, they act. From detection to resolution, they handle incidents and requests instantly, taking your IT management from automated to autonomous.
-
1
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
2
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. -
3
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
4
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
Grafana: The open and composable observability platformGrafana is the open source analytics & monitoring solution for every database.
-
5
Easyspider - Distributed Web Crawler
Easy Spider is a distributed Perl Web Crawler Project from 2006
Easy Spider is a distributed Perl Web Crawler Project from 2006. It features code from crawling webpages, distributing it to a server and generating xml files from it. The client site can be any computer (Windows or Linux) and the Server stores all data. Websites that use EasySpider Crawling for Article Writing Software: https://www.artikelschreiber.com/en/ https://www.unaique.net/en/ https://www.unaique.com/ https://www.artikelschreiben.com/ https://www.buzzerstar.com/ https://easyperlspider.sourceforge.io/ https://www.sebastianenger.com/ https://www.artikelschreiber.com/opensource/ It is fun to look at some code that is few years ago and to see how one has improved himself. ... -
6
File System Crawler for Elasticsearch
Elasticsearch File System Crawler (FS Crawler)
This crawler helps to index binary documents such as PDF, Open Office, MS Office. Local file system (or a mounted drive) crawling and indexing new files, updating existing ones, and removing old ones. Remote file system over SSH/FTP crawling. REST interface to let you “upload” your binary documents to elastic search. -
7
YaCy Peer-to-Peer Search Engine
Decentralized Web Search Engine
YaCy is a free search engine that anyone can use to build search the internet (www and ftp) or to create a search portal for others (internet or intranet). The scale of YaCy is limited only by the number of users and can index billions of web pages. In p2p mode it is fully decentralized, all users of the search engine network are equal and it is not possible for anyone to censor the content of the distributed index. -
8
Vishesh Auto Index
Vishesh Auto Index 5 is Best Wap Auto Index for mp3 sites.
Vishesh Auto Index is a rebirth of AutoIndex. A mobile focused autoindex script that allows users to create a mobile download site. Security level too high. Vishesh Auto Index 5 specially made for mp3/mp4 downloading websites. Vishesh Auto Index 5 My Official Blog:- https://visheshgrewal.blogspot.com Vishesh Auto Index:- https://vishesh-auto-index.blogspot.com Readme.txt for more information -
9
Firing Range
Firing Range is a test bed for web application security scanners
...The project doesn’t just include simple XSS forms; it spans variants such as DOM-based issues, context-sensitive sinks, template mishandling, CSRF, open redirects, and mixed content problems. Each scenario is crafted to reflect how bugs appear in production—behind frameworks, in odd encodings, or across redirects—so scanners must demonstrate accurate crawling and context understanding. Because the behaviors are stable and documented, teams can run comparative tests over time and quantify regression or improvement in their pipelines. It’s equally useful for human training, giving analysts a safe playground to practice exploitation and triage skills. -
Desktop and Mobile Device Management SoftwareDesktop Central is a unified endpoint management (UEM) solution that helps in managing servers, laptops, desktops, smartphones, and tablets from a central location.
-
10
The Teachingbox uses advanced machine learning techniques to relieve developers from the programming of hand-crafted sophisticated behaviors of autonomous agents (such as robots, game players etc...) In the current status we have implemented a well founded reinforcement learning core in Java with many popular usecases, environments, policies and learners. Obtaining the teachingbox: FOR USERS: If you want to download the latest releases, please visit:...
-
11
Excel file analysis toolkit
Lays out dependencies (links) between Excel files in a visual graph
Do you have a haystack (legacy) of Excel files that reference each other and you can't see clear anymore? This tools crawls Excel files and builds up a visual graph. It shows which files might be outdated because they reference a file which was modified more recently. The graph is dynamic, you can choose between different layout algorithms and rearrange by hand. Allows you to quickly open the file, dump the graph, dump all links within a file, highlight all the cells that reference other... -
12
Linkcrawler
Capable to "Crawl" a site and return a report of all links from it
Java Desktop application capable to "Crawl" a site and return a report of the status of all the link present at the page, then it moves to another internal page and so on. LinkCrawlers provides a nice HTML5 report with the information of all link per WebPage, Easy to Read. This tool is useful for Web QA testers -
13
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
14
Egoboo
Free 3D Roguelike action RPG
Egoboo is a working cool 3D dungeon crawling game in the spirit of nethack. Currently it is supported by Windows, Linux and Mac. -
15
Addons for IOSEC - DoS HTTP Security
IOSec Addons are enhancements for web security and crawler detection
...Added Facebook Bot Support for "Facebot/1.0". Add this code to your website to prevent unauthorized stealing of your valuable content & block malicious bots from crawling your site. -
16
druid4arduino
An automatic, configuration-less, GUI for Arduino projects.
Druid4Arduino provides a simple GUI (graphical user interface) to interact with any SerialUI-based Arduino project. It works its magic by crawling the menu hierarchy (commands and sub-menus) provided by SerialUI and automatically re-configuring it’s user interface to match whatever options you’ve provided. It will connect to you’re arduino project through the USB serial port and display a reflection of all the commands and sub-menus defined in your program/sketch. It will also request and transmit any required input or error messages. -
17
LAIR! the game
LAIR is a Sandbox Dungeon Crawling Game
LAIR! is a game designed to provide virtually limitless versatility in the environment that it provides. While you create a new character, it generates a dungeon for you to explore with thousands of creatures, treasures, and secrets. Every piece of content can be generated procedurally by the game engine, so new dungeons and treasures can appear at any time. -
18
Zoozle Search & Download Suchmaschine
Zoozle 2008 - 2010 Webpage, Tools and SQL Files
Download search engine and directory with Rapidshare and Torrent - zoozle Download Suchmaschine All The files that run the World Leading German Download Search Engine in 2010 with 500 000 unique visitors a day - all the tools you need to set up a clone. Code Contains: - PHP Files for zoozle - Perl Crawler for gathering new content to database and all other cool tools i have... -
19
Jurpe
Role Playing Game Engine
JURPE (Java Universal Role Playing Engine) is a Java API to support the writing of new computer RPG games based on a skill based, three dice RPG system. It comes with a full featured game demo of a medieval warrior crawling into a monster filled dungeon -
20
LinkChecker
check links in web documents or full websites
New Homepage: http://wummel.github.io/linkchecker/ Linkchecker features: - recursive and multithreaded checking and site crawling - output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats - HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support - restrict link checking with regular expression filters for URLs - proxy support - username/password authorization for HTTP, FTP -
21
HTTP Anti Flood/DoS Security Module
Detect Flooder IPs, Reduce Attack Surface against HTTP Flood Attacks
This module provides attack surface reduction enhancements against the HTTP Flood Attacks at the web application level. Massive crawling/scanning tools, HTTP Flood tools can be detected and blocked by this module via htaccess, firewall or iptables, etc. (like mod_evasive) You can use this module by including "iosec.php" to any PHP file which wants to be protected. You can test module here: http://www.iosec.org/test.php (demo) Watch the Proof of Concept video: http://goo.gl/dSiAL Hakin9 IT Security Magazine Article about IOSEC http://goo.gl/aQM4Di (different format -> http://goo.gl/JKMUPN) IJNSA Article at http://goo.gl/LLxRdX WP Plugin Page http://goo.gl/nF5nD CHANGES v.1.8.2 - Iptables Auto Ban Bash Script Included - Token Access via Implicit Deny - Reverse Proxy Support - reCAPTCHA Support Do you want more features? ... -
22
Python Crawler Library
Python Web Crawler Library
A simple library for crawling the web. This library will give you the ability to create macros for crawling web site and preforming simple actions like preforming "log in" and other simple actions in web sites. -
23
-
24
Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net
-
25
Law Leecher is a multi-threaded web crawling tool which extracts laws from the EU law database PreLex (http://ec.europa.eu/prelex/). It's written in Ruby.
