Search Results for "web crawler source code"
Sort By:
Showing 1163 open source projects for "web crawler source code"
View related business solutions-
Go From AI Idea to AI App FastAccess Gemini 3 and 200+ models. Build chatbots, agents, or custom models with built-in monitoring and scaling.
-
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
SiteOne Crawler
SiteOne Crawler is a website analyzer and exporter
SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a... -
3
crwlr
Library for Rapid (Web) Crawler and Scraper Development
This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler... -
4
Yii Web Programming Framework
Yii PHP Framework 1.1.x
Yii, a high-performance component-based PHP framework. Yii is a fast, secure, and efficient PHP framework. Flexible yet pragmatic. Works right out of the box. Has reasonable defaults. Yii gives you the maximum functionality by adding the least possible overhead. Sane defaults and built-in tools helps you write solid and secure code. Write more code in less time with simple, yet powerful APIs and code generation. The minimum requirement by Yii is that your Web server supports PHP 5.1.0 or... -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
PHP QR Code
A PHP QR Code generator and reader with a user-friendly API
chillerlan/php-qrcode is a modern, flexible PHP library for generating QR codes. It supports various customization options such as size, encoding, error correction, and logo embedding. The library is PSR-compliant and built for ease of use in modern PHP projects, making it suitable for generating QR codes in web apps, receipts, and authentication systems. -
6
Roach
The complete web scraping toolkit for PHP
Roach is a complete web scraping toolkit for PHP. It is a shameless clone heavily inspired by the popular Scrapy package for Python. Roach allows us to define spiders that crawl and scrape web documents. But wait, there’s more. Roach isn’t just a simple crawler, but includes an entire pipeline to clean, persist and otherwise process extracted data as well. It’s your all-in-one resource for web scraping in PHP. Roach doesn’t depend on a specific framework. Instead, you can use the core... -
7
QueryList
Progressive PHP web crawler framework with jQuery-like DOM parsing
QueryList is an extensible PHP web scraping and crawling framework designed to extract and process data from web pages. It provides a simple and expressive API that allows developers to collect structured information from HTML documents using familiar DOM traversal techniques. It is built on top of phpQuery and uses CSS3 selectors similar to those found in jQuery, making it easy for developers to query and manipulate page elements during scraping tasks. QueryList supports common data... -
8
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across... -
9
Webgrind
Xdebug Profiling Web Frontend in PHP
Webgrind is an Xdebug profiling web frontend for PHP, offering a lightweight and easy-to-use interface to visualize profiling data. It helps developers identify performance bottlenecks in their applications by displaying function call statistics, memory usage, and execution time. Webgrind is ideal for performance analysis and optimization of PHP code, providing insights into code efficiency. -
Forever Free Full-Stack Observability | Grafana CloudBuilt on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
-
10
Laravel Pint
Laravel Pint is an opinionated PHP code style fixer for minimalists.
Laravel Pint is an opinionated PHP code-style fixer for minimalists. Pint is built on top of PHP-CS-Fixer and makes it simple to ensure that your code style stays clean and consistent. Pint is automatically installed with all new Laravel applications so you may start using it immediately. By default, Pint does not require any configuration and will fix code style issues in your code by following the opinionated coding style of Laravel. Pint is included in recent releases of the Laravel... -
11
KodExplorer
A web based file manager, web IDE / browser based code editor
KodExplorer is a file manager for the web. It is also a web code editor, which allows you to develop websites directly within the web browser. You can run KodExplorer either online or locally, on Linux, Windows, or Mac-based platforms. The only requirement is to have PHP 5 available. Use experience like operating system, Rich context menu, and toolbar, drag, and drop, shortcut keys. All operations with files and folders on a remote server (copy, cut, paste, move, remove, upload, create... -
12
PrestaShop
An open source eCommerce platform
...See all the available features. This repository contains the source code of PrestaShop, which is intended for development and preview only. To download the latest stable public version of PrestaShop (currently, version 1.7), please go to the download page on the official PrestaShop site. The 'develop' branch of this repository contains the work in progress source code for the next version of PrestaShop 1.7. -
13
phpMyAdmin
A web interface for MySQL and MariaDB
phpMyAdmin is a free software tool written in PHP, intended to handle the administration of MySQL over the Web. phpMyAdmin supports a wide range of operations on MySQL and MariaDB. Frequently used operations (managing databases, tables, columns, relations, indexes, users, permissions, etc) can be performed via the user interface, while you still have the ability to directly execute any SQL statement. phpMyAdmin comes with a wide range of documentation and users are welcome to update our wiki... -
14
Bloxtor framework
Bloxtor is a professional and intelligent framework
Bloxtor is a professional and intelligent web-development framework designed for No-Coders, Low-Coders, and Coders alike. It functions as a versatile code reader and converter, seamlessly transforming code into visual environments and visual components back into code. The concept is to provide a centralized AI-powered web development platform with multiple levels of Citizen Development, empowering both technical and non-technical developers to easily create any type of web application, while... -
15
Laravel Framework
Elegant PHP web application framework with expressive syntax
Laravel is a popular open-source PHP web framework designed for building modern web applications with an elegant and readable syntax. It emphasizes developer productivity, offering features like routing, Eloquent ORM, blade templating, and an extensive ecosystem of packages. Laravel makes common tasks like authentication, caching, and session management simple and intuitive, making it a top choice for PHP developers. -
16
Timber themes
Create WordPress themes with OOP code and the Twig template engine
Timber helps you create fully-customized WordPress themes faster with more sustainable code. With Timber, you write your HTML using the Twig Template Engine separate from your PHP files. This cleans up your theme code so, for example, your PHP file can focus on being the data/logic, while your Twig file can focus 100% on the HTML and display. Once Timber is installed and activated in your plugins directory, it gives any WordPress theme the ability to take advantage of the power of Twig and... -
17
multiOTP open source
PHP strong authentication library, web interface & CLI, OATH certified
multiOTP is a PHP class, a powerful command line utility and a web interface developed by SysCo systèmes de communication sa in order to provide a completely free and easy operating system independent server side implementation for strong two factors authentication solution. multiOTP supports hardware and software tokens with different One-Time Password algorithms like OATH/HOTP, OATH/TOTP and mOTP (Mobile-OTP). QRcode generation is also embedded in order to support provisioning of Google... -
18
webERP Accounting & Business Management
Accounting & Best Practice Business Administration System
Entirely web-based. It is coded in efficient PHP to present a very low overhead on hardware presenting screens fast & capable of working over low bandwidth connections. Now with HTML 5 & javascript form validation & table sorting. Mature codebase with a wealth of features: - Multi-currency - Multi-inventory location - Serial and Lot tracking inventory - Full double entry general ledger - Weighted Average or Standard Costing - Manufacturing works orders - Material Requirements... -
19
PHP dotenv
Loads environment variables automatically
You should never store sensitive credentials in your code. Storing configuration in the environment is one of the tenets of a twelve-factor app. Anything that is likely to change between deployment environments, such as database credentials or credentials for 3rd party services, should be extracted from the code into environment variables. Basically, a .env file is an easy way to load custom configuration variables that your application needs without having to modify .htaccess files or... -
20
Package Generator
Generates a PHP SDK based on a WSDL, simple and powerful, WSDL to PHP
Package Generator generates a PHP SDK from any WSDL so you can easily consume any SOAP Web Service without wondering how SOAP is used under the hood. Package Generator provides many options to generate your package even if a few are required. This project has been tested with many WSDL and is currently used on the platform Providr.IO. Package Generator generates files that are detailed in the MANIFEST. You are encouraged to read it to understand how and why the files are generated in... -
21
skycaiji
Open source web scraping system for automated data collection tasks
SkyCaiji is an open source web scraping and data collection system designed to gather information from websites through configurable extraction rules. It focuses on simplifying the process of building crawlers by allowing users to visually define scraping rules rather than writing complex code. It can collect structured or unstructured data from many types of webpages and automate the extraction process for large datasets. -
22
Yii 2
The Fast, Secure and Professional PHP Framework
Yii is a fast, secure, and efficient PHP framework. Flexible yet pragmatic. Works right out of the box. Has reasonable defaults. Yii gives you the maximum functionality by adding the least possible overhead. Sane defaults and built-in tools helps you write solid and secure code. Write more code in less time with simple, yet powerful APIs and code generation. It is fast, secure and efficient and works right out of the box pre-configured with reasonable defaults. The framework is easy to... -
23
WordPress
Just a mirror of the WordPress subversion repository
WordPress is one of the world’s most widely used content management systems (CMS), powering blogs, websites, and increasingly web apps. It offers a flexible architecture of themes and plugins, where users can extend functionality or customize layout without touching core code. The administrative dashboard includes post and page editors, media library, user roles, plugin/theme installation, and site settings. Through its REST API and headless mode, WordPress also serves as a backend for... -
24
Testcontainers for PHP
Implementation for PHP
Testcontainers is a PHP package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The package is inspired by the Testcontainers project for Java. Use a containerized instance of your database to test your data access layer code for complete compatibility, without requiring a complex setup on developer machines. Trust that your tests will always start with a known state. Use containerized web browsers, compatible with Selenium, to... -
25
Zenario
Zenario is a web-based content management system (CMS)
Zenario is a web-based content management system (CMS). It can be used for simple sites, with many "wysiwyg" features for making regular web pages, news items, blogs, and so on. It has powerful features for running extranet sites, such as customer portals, and online databases (e.g. of products, documents or videos). It also has multilingual features built in from the core, so that a site can easily be set up to deliver content in in multiple languages.
