Search Results for "web crawler source code"
Sort By:
Showing 315 open source projects for "web crawler source code"
View related business solutions-
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
crawler
Collection of JS reverse engineering examples for web scraping study
crawler is a collection of web scraping and JavaScript reverse engineering examples designed for learning how modern websites protect their data and how those protections can be analyzed. It contains many case studies that demonstrate how to analyze and replicate request parameters, cookies, and encryption logic used by real websites. Each directory in the project focuses on a specific target service or scenario, showing how browser network requests and JavaScript code can be studied to reproduce API calls programmatically. ... -
2
EasySpider
A visual no-code/code-free web crawler/spider
A visual code-free/no-code web crawler/spider, supporting both Chinese and English. -
3
Web-Maker
A blazing fast & offline frontend playground
Web-Maker is an offline playground for your web experiments. Something like CodePen or JSFiddle, but much more faster and works offline because it runs completely on your system. Supports Preprocessors: HTML (Pug & Markdown), CSS (SCSS, LESS & Stylus, Atomic CSS) & JavaScript (ES6, TypeScript & CoffeeScript). Hi! I am Kushagra Gour. Web Maker is a free and open-source project. To keep me motivated for working on such open-source and free side projects, I have launched a Patreon campaign.... -
4
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Katana is an open source command-line web crawling and spidering framework developed by ProjectDiscovery. It is designed to efficiently crawl websites and web applications in order to discover endpoints, resources, and other useful information that may not be easily visible through manual browsing. Katana focuses on speed and automation, making it suitable for use in security reconnaissance workflows and automated pipelines. Katana supports both standard HTTP crawling and headless browser... -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
5
IPFS Web UI
A frontend for an IPFS node
A web interface to IPFS, shipped with Kubo, and ipfs-desktop. Check on your node stats, explore the IPLD powered merkle forest, see peers around the world and manage your files, without needing to touch the CLI. When working on the code, run an ipfs daemon, the local dev server, the unit tests, and the storybook component viewer and see the results of your changes as you save files. If you need to run IPFS in a Docker container, you can still have Web UI available by exposing both the... -
6
Playwright Skill for Claude Code
Claude Code Skill for browser automation with Playwright
...The system supports a wide range of use cases, including testing web applications, validating user interfaces, automating workflows, and extracting data from websites. One of its key advantages is its ability to generate custom Playwright code tailored to each request, allowing flexible and context-aware automation. -
7
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
8
web-extension-starter
Web Extension starter to build "Write Once Run on Any Browser"
Web Extension starter to build "Write Once Run on Any Browser" extension. Cross Browser Support (Web-Extensions API). Browser Tailored Manifest generation. Automatic build on code changes. Auto packs browser-specific build files. SASS styling. TypeScript by default. ES6 modules support. React UI Library by default. Smart reload. Used by extensions in production that has over 100,000+ users. -
9
fess
Open source enterprise search server for websites, files, and data
...Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. It supports indexing and searching across many document formats including office documents, PDFs, and compressed archives. It also provides a web-based administrative interface that allows administrators to configure crawling targets, manage indexing tasks, and adjust search settings from a graphical dashboard. -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
10
Agregore Browser
A minimal browser for the distributed web (Desktop version)
A minimal web browser for the distributed web. Web Extension support. Built-in Markdown/Gemini/JSON rendering extension. Built-in QR code scanner and generator extension. Generate a QR code for the current page. Scan a QR code from the browser action window. Right-click a link or image to generate a QR code for it. Built-in ad blocker (ublock origin). Built-in support for creating web archives via ArchiveWeb.page. -
11
Stylus for Chrome
Stylus - Userstyles Manager
Stylus is a fork of Stylish for Chrome, also compatible with Firefox as a WebExtension. "Stylus" is a fork of the popular Stylish extension which can be used to restyle the web. Not "ish", but "us", as in "us" the actual users. Stylus is a fork of Stylish that is based on the source code of version 1.5.2, which was the most up-to-date version before the original developer stopped working on the project. The objective in creating Stylus was to remove any and all analytics, and return to a more user-friendly UI. ... -
12
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across... -
13
Ghostery
Ghostery Browser Extension for Firefox, Chrome, Opera and Edge
Ghostery helps you browse smarter by giving you control over ads and tracking technologies to speed up page loads, eliminate clutter, and protect your data. This is the unified code repository for the Ghostery browser extensions in Chrome, Firefox, Opera and Edge. Browse the web safer, faster & with less annoying ads. Equipped with award-winning AI anti-tracking technology to browse the websafe and quickly. Ghostery helps you stay informed about what companies are tracking you by listing the... -
14
MetaMask
Mobile web browser providing access to websites that use the Ethereum
MetaMask is a mobile wallet that provides easy access to websites that use the Ethereum blockchain. A crypto wallet & gateway to blockchain apps. Start exploring blockchain applications in seconds. Trusted by over 30 million users worldwide. Available as a browser extension and as a mobile app, MetaMask equips you with a key vault, secure login, token wallet, and token exchange—everything you need to manage your digital assets. MetaMask provides the simplest yet most secure way to connect to... -
15
Return YouTube Dislike
Chrome extension to return youtube dislikes
Return YouTube Dislike is an open-source extension that returns the YouTube dislike count. Available for Chrome and Firefox as a Web Extension. Also available for other browsers as JS Userscript. Additionally, the dislike field in the YouTube API was removed on December 13th, 2021, removing any ability to judge the quality of content before watching. With the removal of dislike stats from the YouTube API, our backend switched to using a combination of scraped dislike stats, and estimates extrapolated from extension user data. ... -
16
Serverless Express by Vendia
Run Node.js web applications and APIs using existing frameworks
Run REST APIs and other web applications using your existing Node.js application framework (Express, Koa, Hapi, Sails, etc.), on top of AWS Lambda and Amazon API Gateway. Vendia is the real-time data cloud for rapidly building applications that securely share data across departments, companies, clouds, and regions. We’re excited to announce the latest release of Vendia Share! This release includes new features like smart contracts, user-level transactions, the beta release of Azure support,... -
17
Betterfox
Firefox user.js for optimal privacy and security
Betterfox is an opinionated configuration profile for Mozilla Firefox designed to improve everyday web browsing by making the browser faster, more private, and more secure without relying on external add-ons or third-party code. Rather than being a separate browser, it consists of a curated set of preference tweaks (user.js settings) that users apply to their Firefox profile to optimize performance and harden privacy settings. The project focuses on a minimal-impact, maximum-effect approach,... -
18
ego (lite)
The fastest browser for AI agents to run web automation
Ego Lite is a macOS browser designed for people and external AI agents to work in parallel. Each agent receives an isolated Space, allowing several browser tasks to run simultaneously without taking over the user's active tabs. Optional Chrome data migration carries over logins, cookies, extensions, and bookmarks so agents can use authenticated sessions with less setup. The ego-browser skill connects tools such as Codex, Claude Code, Cursor, and custom agent CLIs to the browser. Agents... -
19
Decap
A Git-based CMS for Static Site Generators
Open source content management for your Git workflow. Use Decap CMS with any static site generator for a faster and more flexible web project. Get the speed, security, and scalability of a static site, while still providing a convenient editing interface for content. Content is stored in your Git repository alongside your code for easier versioning, multi-channel publishing, and the option to handle content updates directly in Git. -
20
Jimp
An image processing library written entirely in JavaScript for Node
An image processing library for Node written entirely in JavaScript, with zero native dependencies. If you're using this library with TypeScript the method of importing slightly differs from JavaScript. Instead of using require, you must import it with ES6 default import scheme. If you're using a web bundles (webpack, rollup, parcel) you can benefit from using the module build of jimp. Using the module build will allow your bundler to understand your code better and exclude things you aren't... -
21
Middy
The stylish Node.js middleware engine for AWS Lambda
Middy is a very simple middleware engine that allows you to simplify your AWS Lambda code when using Node.js. If you have used web frameworks like Express, then you will be familiar with the concepts adopted in Middy and you will be able to get started very quickly. A middleware engine allows you to focus on the strict business logic of your Lambda and then attach additional common elements like authentication, authorization, validation, serialization, etc. in a modular and reusable way by... -
22
SPX
A simple & straight-to-the-point PHP profiling extension
SPX, which stands for Simple Profiling eXtension, is just another profiling extension for PHP. It differentiates itself from other similar extensions as being totally free and confined to your infrastructure (i.e. no data leaks to a SaaS). Very simple to use: just set an environment variable (command line) or switch on a radio button (web request) to profile your script. Thus, you are free of manually instrumenting your code (Ctrl-C a long running command line script is even supported).... -
23
Netlify CMS
A Git-based CMS for static site generators
Open source content management for your Git workflow. Use Netlify CMS with any static site generator for a faster and more flexible web project. Get the speed, security, and scalability of a static site, while still providing a convenient editing interface for content. Content is stored in your Git repository alongside your code for easier versioning, multi-channel publishing, and the option to handle content updates directly in Git. -
24
rebroswer-patches
Patches for Puppeteer and Playwright to reduce automation detection
rebrowser-patches is an open source collection of patches designed to improve the stealth capabilities of browser automation frameworks. It focuses primarily on enhancing Puppeteer and Playwright by modifying parts of their source code that may reveal automation activity to websites. Many modern websites rely on bot detection mechanisms that identify automation through behavioral or technical signals, and these patches aim to reduce those detection vectors. By applying targeted fixes, the... -
25
GhostText
Use your text editor to write in your browser
Whenever you’re writing more than a little snippet of code anywhere on the web, activate GhostText to open your preferred text editor and enjoy your own development environment. GhostText is a browser extension that connects to your editor via its own extension. Install both extensions and, if necessary, start the GhostText server in the editor’s extension. Most editor extensions are authored by third parties. You can create more extensions for your favorite editor! Refer to the protocol...
