Search Results for "crawling"
Sort By:
31 projects for "crawling" with 1 filter applied:
-
Vibes don’t ship, Retool doesVibe coding tools create cool demos, but Retool helps you build software your company can actually use. Generate internal apps that connect directly to your data—deployed in your cloud with enterprise security from day one. Build dashboards, admin panels, and workflows with granular permissions already in place. Stop prototyping and ship on a platform that actually passes security review.
-
Atera all-in-one platform IT management software with AI agentsAtera’s AI agents don’t just assist, they act. From detection to resolution, they handle incidents and requests instantly, taking your IT management from automated to autonomous.
-
1
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
2
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
3
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
4
Easyspider - Distributed Web Crawler
Easy Spider is a distributed Perl Web Crawler Project from 2006
Easy Spider is a distributed Perl Web Crawler Project from 2006. It features code from crawling webpages, distributing it to a server and generating xml files from it. The client site can be any computer (Windows or Linux) and the Server stores all data. Websites that use EasySpider Crawling for Article Writing Software: https://www.artikelschreiber.com/en/ https://www.unaique.net/en/ https://www.unaique.com/ https://www.artikelschreiben.com/ https://www.buzzerstar.com/ https://easyperlspider.sourceforge.io/ https://www.sebastianenger.com/ https://www.artikelschreiber.com/opensource/ It is fun to look at some code that is few years ago and to see how one has improved himself. ... -
Find Hidden Risks in Windows Task SchedulerWindows Task Scheduler might be hiding critical failures. Download the free JAMS diagnostic tool to uncover problems before they impact production—get a color-coded risk report with clear remediation steps in minutes.
-
5
File System Crawler for Elasticsearch
Elasticsearch File System Crawler (FS Crawler)
This crawler helps to index binary documents such as PDF, Open Office, MS Office. Local file system (or a mounted drive) crawling and indexing new files, updating existing ones, and removing old ones. Remote file system over SSH/FTP crawling. REST interface to let you “upload” your binary documents to elastic search. -
6
YaCy Peer-to-Peer Search Engine
Decentralized Web Search Engine
YaCy is a free search engine that anyone can use to build search the internet (www and ftp) or to create a search portal for others (internet or intranet). The scale of YaCy is limited only by the number of users and can index billions of web pages. In p2p mode it is fully decentralized, all users of the search engine network are equal and it is not possible for anyone to censor the content of the distributed index. -
7
Vishesh Auto Index
Vishesh Auto Index 5 is Best Wap Auto Index for mp3 sites.
Vishesh Auto Index is a rebirth of AutoIndex. A mobile focused autoindex script that allows users to create a mobile download site. Security level too high. Vishesh Auto Index 5 specially made for mp3/mp4 downloading websites. Vishesh Auto Index 5 My Official Blog:- https://visheshgrewal.blogspot.com Vishesh Auto Index:- https://vishesh-auto-index.blogspot.com Readme.txt for more information -
8
Firing Range
Firing Range is a test bed for web application security scanners
...The project doesn’t just include simple XSS forms; it spans variants such as DOM-based issues, context-sensitive sinks, template mishandling, CSRF, open redirects, and mixed content problems. Each scenario is crafted to reflect how bugs appear in production—behind frameworks, in odd encodings, or across redirects—so scanners must demonstrate accurate crawling and context understanding. Because the behaviors are stable and documented, teams can run comparative tests over time and quantify regression or improvement in their pipelines. It’s equally useful for human training, giving analysts a safe playground to practice exploitation and triage skills. -
9
The Teachingbox uses advanced machine learning techniques to relieve developers from the programming of hand-crafted sophisticated behaviors of autonomous agents (such as robots, game players etc...) In the current status we have implemented a well founded reinforcement learning core in Java with many popular usecases, environments, policies and learners. Obtaining the teachingbox: FOR USERS: If you want to download the latest releases, please visit:...
-
Reach Your Audience with Rise Vision, the #1 Cloud Digital Signage Software SolutionRise Vision is the #1 digital signage company, offering easy-to-use cloud digital signage software compatible with any player across multiple screens. Forget about static displays. Save time and boost sales with 500+ customizable content templates for your screens. If you ever need help, get free training and exceptionally fast support.
-
10
Excel file analysis toolkit
Lays out dependencies (links) between Excel files in a visual graph
Do you have a haystack (legacy) of Excel files that reference each other and you can't see clear anymore? This tools crawls Excel files and builds up a visual graph. It shows which files might be outdated because they reference a file which was modified more recently. The graph is dynamic, you can choose between different layout algorithms and rearrange by hand. Allows you to quickly open the file, dump the graph, dump all links within a file, highlight all the cells that reference other... -
11
Linkcrawler
Capable to "Crawl" a site and return a report of all links from it
Java Desktop application capable to "Crawl" a site and return a report of the status of all the link present at the page, then it moves to another internal page and so on. LinkCrawlers provides a nice HTML5 report with the information of all link per WebPage, Easy to Read. This tool is useful for Web QA testers -
12
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
13
Addons for IOSEC - DoS HTTP Security
IOSec Addons are enhancements for web security and crawler detection
...Added Facebook Bot Support for "Facebot/1.0". Add this code to your website to prevent unauthorized stealing of your valuable content & block malicious bots from crawling your site. -
14
druid4arduino
An automatic, configuration-less, GUI for Arduino projects.
Druid4Arduino provides a simple GUI (graphical user interface) to interact with any SerialUI-based Arduino project. It works its magic by crawling the menu hierarchy (commands and sub-menus) provided by SerialUI and automatically re-configuring it’s user interface to match whatever options you’ve provided. It will connect to you’re arduino project through the USB serial port and display a reflection of all the commands and sub-menus defined in your program/sketch. It will also request and transmit any required input or error messages. -
15
Jurpe
Role Playing Game Engine
JURPE (Java Universal Role Playing Engine) is a Java API to support the writing of new computer RPG games based on a skill based, three dice RPG system. It comes with a full featured game demo of a medieval warrior crawling into a monster filled dungeon -
16
LinkChecker
check links in web documents or full websites
New Homepage: http://wummel.github.io/linkchecker/ Linkchecker features: - recursive and multithreaded checking and site crawling - output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats - HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support - restrict link checking with regular expression filters for URLs - proxy support - username/password authorization for HTTP, FTP -
17
HTTP Anti Flood/DoS Security Module
Detect Flooder IPs, Reduce Attack Surface against HTTP Flood Attacks
This module provides attack surface reduction enhancements against the HTTP Flood Attacks at the web application level. Massive crawling/scanning tools, HTTP Flood tools can be detected and blocked by this module via htaccess, firewall or iptables, etc. (like mod_evasive) You can use this module by including "iosec.php" to any PHP file which wants to be protected. You can test module here: http://www.iosec.org/test.php (demo) Watch the Proof of Concept video: http://goo.gl/dSiAL Hakin9 IT Security Magazine Article about IOSEC http://goo.gl/aQM4Di (different format -> http://goo.gl/JKMUPN) IJNSA Article at http://goo.gl/LLxRdX WP Plugin Page http://goo.gl/nF5nD CHANGES v.1.8.2 - Iptables Auto Ban Bash Script Included - Token Access via Implicit Deny - Reverse Proxy Support - reCAPTCHA Support Do you want more features? ... -
18
Python Crawler Library
Python Web Crawler Library
A simple library for crawling the web. This library will give you the ability to create macros for crawling web site and preforming simple actions like preforming "log in" and other simple actions in web sites. -
19
Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net
-
20
Law Leecher is a multi-threaded web crawling tool which extracts laws from the EU law database PreLex (http://ec.europa.eu/prelex/). It's written in Ruby.
-
21
An PHP + MySQL script that attempts to recreate features similar to the likes of Mininova, SeedPeer, and TPB frontend. Features a new design, crawling ability, cURL for single torrent scrapes + mass scrapes, and much much more! Based on T-Xore 0.4, Ibitz
-
22
A cross-platform, highly configurable, distributed web crawling system with crawlers optimized for speed.
-
23
This is a game that is *sort of* in the tetris puzzle style. Various types of ants are crawling along and you have to swat them into the pants so the picnic food won't get stolen. Windows and Linux Versions now available.
-
24
Ruya is a Python-based breadth-first, level-, delayed, event-based-crawler for crawling English, Japanese websites. It is targeted solely towards developers who want crawling functionality in their projects using API, and crawl control.
-
25
MPI implementation using PHP, project still under development... This will give you a speed-up if your code relies alot on bandwidth(eg: crawling the web), i think its useless if your program doesnt require internet.
