Search Results for "python web crawler"
Sort By:
Showing 170 open source projects for "python web crawler"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
crwlr
Library for Rapid (Web) Crawler and Scraper Development
...Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could just load actually all links it is finding (and is allowed to load according to the robots.txt file), then it would just load the whole internet (if the URL(s) it starts with are no dead end). Or it can be restricted to load only links matching certain criteria (on same domain/host, URL path starts with "/foo",...) or only to a certain depth. ... -
2
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across... -
3
Roach
The complete web scraping toolkit for PHP
Roach is a complete web scraping toolkit for PHP. It is a shameless clone heavily inspired by the popular Scrapy package for Python. Roach allows us to define spiders that crawl and scrape web documents. But wait, there’s more. Roach isn’t just a simple crawler, but includes an entire pipeline to clean, persist and otherwise process extracted data as well. -
4
QueryList
Progressive PHP web crawler framework with jQuery-like DOM parsing
QueryList is an extensible PHP web scraping and crawling framework designed to extract and process data from web pages. It provides a simple and expressive API that allows developers to collect structured information from HTML documents using familiar DOM traversal techniques. It is built on top of phpQuery and uses CSS3 selectors similar to those found in jQuery, making it easy for developers to query and manipulate page elements during scraping tasks. QueryList supports common data... -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
Seeker
Accurately Locate Smartphones using Social Engineering
Seeker is an open source project that demonstrates how to obtain precise location information from devices using social engineering and web-based techniques. The tool sets up a phishing page that asks for location permissions, allowing GPS and other device data to be shared if the user consents. It can capture latitude, longitude, accuracy, altitude, direction, and even speed, with results displayed in a terminal. The project supports both manual deployment and tunneling services like Ngrok... -
6
Requests for PHP
Requests for PHP is a humble HTTP request library
Requests is a HTTP library written in PHP, for human beings. It is roughly based on the API from the excellent Requests Python library. Requests is ISC Licensed (similar to the new BSD license) and has no dependencies, except for PHP 5.6+. Despite PHP’s use as a language for the web, its tools for sending HTTP requests are severely lacking. cURL has an interesting API, to say the least, and you can’t always rely on it being available. Sockets provide only low-level access and require you to build most of the HTTP response parsing yourself. ... -
7
Ascoos Web Extended Studio
Is a portable web server suite for windows 64Bit, for Web Development.
Ascoos Web Extended Studio (AWES) is a portable, free 64-bit web server environment for Windows, designed for professional web developers and designers who need flexibility, modularity, and multi-version testing capabilities. It provides a complete local development stack based on technologies such as Apache, PHP, Node.js, Python, MariaDB, MongoDB, FileZilla, and other essential tools -
8
ahCrawler
A PHP search engine for your website and web analytics tool. GNU GPL3
ahCrawler is a set to implement your own search on your website and an analyzer for your web content. It can be used on a shared hosting. It consists of * crawler (spider) and indexer * search for your website(s) * search statistics * website analyzer (http header, short titles and keywords, linkchecker, ...) You need to install it on your own server. So all crawled data stay in your environment. You never know when an external webspider updated your content. ... -
9
Network Security Toolkit (NST)
A network security analysis and monitoring toolkit Linux distribution.
Network Security Toolkit (NST) is a bootable ISO image (Live USB Flash Drive) based on Fedora 44 providing easy access to best-of-breed Open Source Network Security Applications and should run on most x86_64 systems. The main intent of developing this toolkit was to provide the security professional and network administrator with a comprehensive set of Open Source Network Security Tools. The majority of tools published in the article: Top 125 Security Tools by INSECURE.ORG are available... -
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
10
CacheGuard Gateway
Free UTM appliance: firewall, VPN, WAF and antivirus in one ISO.
Securing your network should not require an enterprise budget. CacheGuard is a free open-source network security appliance for startups and growing businesses that need serious protection without the complexity. Install CacheGuard-OS on any x86 machine or VM and get a complete security gateway in under an hour. No plug-ins, no compatibility issues. Everything works out of the box. CacheGuard-OS is not an app, it IS the OS. A fully custom network appliance operating system built from... -
11
LocNetServe
A local web server for developing web applications on Windows.
LocNetServe is a web development platform for Windows that helps you build dynamic web applications using Apache 2.4, PHP 8.2, and MySQL 9.4. It installs everything you need to develop web applications easily, thanks to its modern interface. You can manage your server and develop applications without manually editing configuration files. Best of all, LocNetServe is free (under MIT license) and available in a 64-bit version designed for modern Windows systems. -
12
elFinder is a file manager for web similar to that you use on your computer. Written in JavaScript using jQuery UI, it just work's in any modern browser. Its creation is inspired by simplicity and convenience of Finder.app program used in Mac OS X.
-
13
Ascoos Web Server
Is a web server for all Web Developers and Web Designers
For PHP 5.6 - 8.4.X see: Ascoos Web Extended Studio (AWES) is here : https://sourceforge.net/projects/ascoos-web-extended-studio/ ASCOOS Web Server is a rich package designed as a versatile web server for development purposes. It incorporates third-party components such as PHP, MySQL, pgSQL, MongoDB and FileZilla and stands out through a compact setup and a well-built administrative panel. ASCOOS Web Server allows you to work with multiple versions of PHP and MySQL without having to... -
14
CerberusCMS5
Cerberus Content Management System
Cerberus Content Management System is a dynamic, secure and infinitely expandable CMS designed after a Unix-Like model. It is a custom written Web Application Framework ( W.A.F. ) with a consistent and custom written Pre-Hyper-Text-Post-Processor Programming Code Framework ( P.C.F. ). This Web Application Software Project' aim is to be the fastest and most secure Web Application Framework, Web Application Programming Code Framework, Text, Voice and Video Communications Platform and Content... -
15
Crawlab
Distributed web crawler admin platform for spiders management
Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. ... -
16
Goutte
Goutte, a simple PHP Web Scraper
...The method returns a Crawler object (Symfony\Component\DomCrawler\Crawler). To use your own HTTP settings, you may create and pass an HttpClient instance to Goutte. For example, to add a 60 second request timeout. Read the documentation of the BrowserKit, DomCrawler, and HttpClient Symfony Components for more information about what you can do with Goutte. Goutte is a thin wrapper around the following Symfony Components: BrowserKit, CssSelector, DomCrawler, and HttpClient. -
17
FormaVid
Small Business Appliance
The FormaVid Small Business Appliance https://formavid.org is designed to integrate a content management system (CMS), an issues tracker and an invoicing application into a single, well constructed, offering. It is an excellent starting point for any developer(s) wishing to support the CMS or any of the other components, including the appliance itself. All components are stable, open source and well supported. The appliance is built using scripts so no hidden "monkey business" and you can... -
18
SabroCMS CMS and Online FTP
SABROCMS (CONTENT MANAGEMENT SYSTEM) PHP SCRIPT TO EDIT WEBSITES
SabroCMS is a Freemium PHP Script that allows you to manage the files of your hosting or your website, from any web browser, either from your computer, tablet or cell phone, with online FTP functions, ideal for uploading files, deleting files, renaming files, clone files, create or delete directories and especially to create and edit files online, such as HTML files, Javascript JS files, CSS Style Sheets, Python PT Scripts, PHP Scripts, Perl, CGI and all kinds of TXT, XML, and more! -
19
Liferay Portal
The world's leading open source portal
Liferay Portal is the world's leading enterprise open source portal framework, offering integrated Web publishing and content management, an enterprise service bus and service-oriented architecture, and compatibility with all major IT infrastructure. Check GitHub for our latest releases: https://github.com/liferay/liferay-portal/releases https://github.com/liferay/liferay-ide/releases -
20
KeexyBox
KeexyBox has moved - Historical Raspberry Pi Project
KeexyBox has moved. This open-source project is now archived. Today, our mission is to help families create a safer and healthier digital life. KeexyBox is now a simple web-based parental Internet protection platform that helps parents protect children online, manage Internet access, and build better digital habits for the whole family—without dedicated hardware. 👉 Learn more: https://keexybox.com -
21
RED HAWK
All-in-one reconnaissance and vulnerability scanning toolkit for sites
RED HAWK is an open source command-line security tool designed for information gathering, vulnerability scanning, and web reconnaissance tasks. It combines multiple scanning and analysis capabilities into a single toolkit to help security researchers and penetration testers quickly analyze a target website. It can collect a wide range of information about domains, servers, and web applications, including network details, hosting configuration, and content management system detection. It also... -
22
PHP mini vulnerability suite
Multiple server/webapp vulnerability scanner
github: https://github.com/samedog/phpmvs -
23
-
24
gdpr
Tool to maintain gdpr data protection declaration
Admins often maintain multiple web pages, each of which under EU-GDPR requires a privacy statement. In order to keep them coherent, up-to-date and at the same time avoiding doing the same work multiple times, this project provides a tool to automatically create the appropriate statements for each page from a single source. The project is currently available in PHP, however if anyone is willing to provide a version in Python or Perl or whatever, it is more than welcome. ... -
25
OpenSearchServer Search Engine
An open source search engine with RESTFul API and crawlers
OpenSearchServer is a powerful, enterprise-class, search engine program. Using the web user interface, the crawlers (web, file, database, etc.) and the client libraries (REST/API , Ruby, Rails, Node.js, PHP, Perl) you will be able to integrate quickly and easily advanced full-text search capabilities in your application: Full-text with basic semantic, join queries, boolean queries, facet and filter, document (PDF, Office, etc.) indexation, web scrapping,etc. OpenSearchServer runs on...
