Search Results for "python web crawler"
Sort By:
Showing 165 open source projects for "python web crawler"
View related business solutions-
Cloud-based help desk software with ServoDeskWhat if You Could Automate 90% of Your Repetitive Tasks in Under 30 Days? At ServoDesk, we help businesses like yours automate operations with AI, allowing you to cut service times in half and increase productivity by 25% - without hiring more staff.
-
Next-Gen Encryption for Post-Quantum Security | CLEAR by Quantum KnightCLEAR by Quantum Knight is a FIPS-140-3 validated encryption SDK engineered for enterprises requiring top-tier security. Offering robust post-quantum cryptography, CLEAR secures files, streaming media, databases, and networks with ease across over 30 modern platforms. Its compact design, smaller than a single smartphone image, ensures maximum efficiency and low energy consumption.
-
1
crwlr
Library for Rapid (Web) Crawler and Scraper Development
...Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could just load actually all links it is finding (and is allowed to load according to the robots.txt file), then it would just load the whole internet (if the URL(s) it starts with are no dead end). Or it can be restricted to load only links matching certain criteria (on same domain/host, URL path starts with "/foo",...) or only to a certain depth. ... -
2
Roach
The complete web scraping toolkit for PHP
Roach is a complete web scraping toolkit for PHP. It is a shameless clone heavily inspired by the popular Scrapy package for Python. Roach allows us to define spiders that crawl and scrape web documents. But wait, there’s more. Roach isn’t just a simple crawler, but includes an entire pipeline to clean, persist and otherwise process extracted data as well. -
3
Crawlab
Distributed web crawler admin platform for spiders management
Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. ... -
4
Requests for PHP
Requests for PHP is a humble HTTP request library
Requests is a HTTP library written in PHP, for human beings. It is roughly based on the API from the excellent Requests Python library. Requests is ISC Licensed (similar to the new BSD license) and has no dependencies, except for PHP 5.6+. Despite PHP’s use as a language for the web, its tools for sending HTTP requests are severely lacking. cURL has an interesting API, to say the least, and you can’t always rely on it being available. Sockets provide only low-level access and require you to build most of the HTTP response parsing yourself. ... -
Free and Open Source HR SoftwareGive your HR team the tools they need to streamline administrative tasks, support employees, and make informed decisions with the OrangeHRM free and open source HR software.
-
5
ahCrawler
A PHP search engine for your website and web analytics tool. GNU GPL3
ahCrawler is a set to implement your own search on your website and an analyzer for your web content. It can be used on a shared hosting. It consists of * crawler (spider) and indexer * search for your website(s) * search statistics * website analyzer (http header, short titles and keywords, linkchecker, ...) You need to install it on your own server. So all crawled data stay in your environment. You never know when an external webspider updated your content. ... -
6
Ascoos Web Server
Is a web server for all Web Developers and Web Designers
For PHP 5.6 - 8.4.X see: Ascoos Web Extended Studio (AWES) is here : https://sourceforge.net/projects/ascoos-web-extended-studio/ ASCOOS Web Server is a rich package designed as a versatile web server for development purposes. It incorporates third-party components such as PHP, MySQL, pgSQL, MongoDB and FileZilla and stands out through a compact setup and a well-built administrative panel. ASCOOS Web Server allows you to work with multiple versions of PHP and MySQL without having to... -
7
Ascoos Web Extended Studio
Is a portable web server suite for windows 64Bit, for Web Development.
Ascoos Web Extended Studio (AWES) is a portable, free 64-bit web server environment for Windows, designed for professional web developers and designers who need flexibility, modularity, and multi-version testing capabilities. It provides a complete local development stack based on technologies such as Apache, PHP, Node.js, Python, MariaDB, MongoDB, FileZilla, and other essential tools -
8
Network Security Toolkit (NST)
A network security analysis and monitoring toolkit Linux distribution.
Network Security Toolkit (NST) is a bootable ISO image (Live USB Flash Drive) based on Fedora 42 providing easy access to best-of-breed Open Source Network Security Applications and should run on most x86_64 systems. The main intent of developing this toolkit was to provide the security professional and network administrator with a comprehensive set of Open Source Network Security Tools. The majority of tools published in the article: Top 125 Security Tools by INSECURE.ORG are available... -
9
LocNetServe
A local web server for developing web applications on Windows.
LocNetServe is a web development platform for Windows that helps you build dynamic web applications using Apache 2.4, PHP 8.2, and MySQL 9.4. It installs everything you need to develop web applications easily, thanks to its modern interface. You can manage your server and develop applications without manually editing configuration files. Best of all, LocNetServe is free (under MIT license) and available in a 64-bit version designed for modern Windows systems. -
AestheticsPro Medical Spa SoftwareAestheticsPro is the most complete Aesthetics Software on the market today. HIPAA Cloud Compliant with electronic charting, integrated POS, targeted marketing and results driven reporting; AestheticsPro delivers the tools you need to manage your medical spa business. It is our mission To Provide an All-in-One Cutting Edge Software to the Aesthetics Industry.
-
10
elFinder is a file manager for web similar to that you use on your computer. Written in JavaScript using jQuery UI, it just work's in any modern browser. Its creation is inspired by simplicity and convenience of Finder.app program used in Mac OS X.
-
11
CerberusCMS5
Cerberus Content Management System
Cerberus Content Management System is a dynamic, secure and infinitely expandable CMS designed after a Unix-Like model. It is a custom written Web Application Framework ( W.A.F. ) with a consistent and custom written Pre-Hyper-Text-Post-Processor Programming Code Framework ( P.C.F. ). This Web Application Software Project' aim is to be the fastest and most secure Web Application Framework, Web Application Programming Code Framework, Text, Voice and Video Communications Platform and Content... -
12
Goutte
Goutte, a simple PHP Web Scraper
...The method returns a Crawler object (Symfony\Component\DomCrawler\Crawler). To use your own HTTP settings, you may create and pass an HttpClient instance to Goutte. For example, to add a 60 second request timeout. Read the documentation of the BrowserKit, DomCrawler, and HttpClient Symfony Components for more information about what you can do with Goutte. Goutte is a thin wrapper around the following Symfony Components: BrowserKit, CssSelector, DomCrawler, and HttpClient. -
13
FormaVid
Small Business Appliance
The FormaVid Small Business Appliance https://formavid.org is designed to integrate a content management system (CMS), an issues tracker and an invoicing application into a single, well constructed, offering. It is an excellent starting point for any developer(s) wishing to support the CMS or any of the other components, including the appliance itself. All components are stable, open source and well supported. The appliance is built using scripts so no hidden "monkey business" and you can... -
14
SabroCMS CMS and Online FTP
SABROCMS (CONTENT MANAGEMENT SYSTEM) PHP SCRIPT TO EDIT WEBSITES
SabroCMS is a Freemium PHP Script that allows you to manage the files of your hosting or your website, from any web browser, either from your computer, tablet or cell phone, with online FTP functions, ideal for uploading files, deleting files, renaming files, clone files, create or delete directories and especially to create and edit files online, such as HTML files, Javascript JS files, CSS Style Sheets, Python PT Scripts, PHP Scripts, Perl, CGI and all kinds of TXT, XML, and more! -
15
Liferay Portal
The world's leading open source portal
Liferay Portal is the world's leading enterprise open source portal framework, offering integrated Web publishing and content management, an enterprise service bus and service-oriented architecture, and compatibility with all major IT infrastructure. Check GitHub for our latest releases: https://github.com/liferay/liferay-portal/releases https://github.com/liferay/liferay-ide/releases -
16
KeexyBox
The box to keep the Internet under your control
KeexyBox allows you to do parental control, block ads, limit telemetry, and browse the Internet anonymously from your home network without installing any software on your devices. It also can be used to create a public wireless access point with the captive portal. It is a software program which requires a Raspberry PI box and which is installed on Raspbian. It constitutes a cut-off point between your devices (computers, tablets, smartphones, etc.) and your router or Internet box.... -
17
PHP mini vulnerability suite
Multiple server/webapp vulnerability scanner
github: https://github.com/samedog/phpmvs -
18
-
19
gdpr
Tool to maintain gdpr data protection declaration
Admins often maintain multiple web pages, each of which under EU-GDPR requires a privacy statement. In order to keep them coherent, up-to-date and at the same time avoiding doing the same work multiple times, this project provides a tool to automatically create the appropriate statements for each page from a single source. The project is currently available in PHP, however if anyone is willing to provide a version in Python or Perl or whatever, it is more than welcome. ... -
20
OpenSearchServer Search Engine
An open source search engine with RESTFul API and crawlers
OpenSearchServer is a powerful, enterprise-class, search engine program. Using the web user interface, the crawlers (web, file, database, etc.) and the client libraries (REST/API , Ruby, Rails, Node.js, PHP, Perl) you will be able to integrate quickly and easily advanced full-text search capabilities in your application: Full-text with basic semantic, join queries, boolean queries, facet and filter, document (PDF, Office, etc.) indexation, web scrapping,etc. OpenSearchServer runs on... -
21
NTK RTMP SERVER
Naam Tamilar Web TV Live Streamer
Naam Tamilar RTMP Server This project updated as open source for future use of Naam Tamilar Political Party. To contribute to the party and in some case if there is any possibilities if i cannot support them for long term. I thought of sharing this source code so in future it may be helpful for the community and party in which other software developers can help them to upgrade. This source is forked from - https://github.com/arut/nginx-rtmp-module and modified with multiple broadcast... -
22
EasyPHP Devserver & Webserver
A complete WAMP environment for PHP development & personal web hosting
EasyPHP Devserver & Webserver Develop with Devserver & Host with Webserver PHP, Apache, MySQL, Nginx, PhpMyAdmin, Xdebug, PostgreSQL, MongoDB, Python, Ruby... for Windows. Devserver installs a complete and ready-to-use development environment. Devserver is portable, modular, fully configurable and easy to update and extend. Webserver turns your computer into a ready-to-use personal web hosting server. You can host whatever you want directly on your computer and share it on internet like any website. ... -
23
OpenWebSpider is an Open Source multi-threaded Web Spider (robot, crawler) and search engine with a lot of interesting features!
-
24
AlienVault OSSIM
Open Source SIEM
OSSIM, AlienVault’s Open Source Security Information and Event Management (SIEM) product, provides event collection, normalization and correlation. For more advanced functionality, AlienVault Unified Security Management (USM) builds on OSSIM with these additional capabilities: * Log management * Advanced threat detection with a continuously updated library of pre-built correlation rules * Actionable threat intelligence updates from AlienVault Labs Security Research Team * Rich... -
25
htmlarea
Small, powerful, full featured WYSIWYG editor
HTMLArea 4 is a browser based WYSIWYG editor that easily replaces the TEXTAREA in your web pages. It is written in JavaScript, and suitable for use in any modern web browser, and any page on your web site. Current version is 4.0-2016-08-29
