Search Results for "web crawler source code"
Sort By:
Showing 313 open source projects for "web crawler source code"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
1
crawler
Collection of JS reverse engineering examples for web scraping study
crawler is a collection of web scraping and JavaScript reverse engineering examples designed for learning how modern websites protect their data and how those protections can be analyzed. It contains many case studies that demonstrate how to analyze and replicate request parameters, cookies, and encryption logic used by real websites. Each directory in the project focuses on a specific target service or scenario, showing how browser network requests and JavaScript code can be studied to reproduce API calls programmatically. ... -
2
EasySpider
A visual no-code/code-free web crawler/spider
A visual code-free/no-code web crawler/spider, supporting both Chinese and English. -
3
Web-Maker
A blazing fast & offline frontend playground
Web-Maker is an offline playground for your web experiments. Something like CodePen or JSFiddle, but much more faster and works offline because it runs completely on your system. Supports Preprocessors: HTML (Pug & Markdown), CSS (SCSS, LESS & Stylus, Atomic CSS) & JavaScript (ES6, TypeScript & CoffeeScript). Hi! I am Kushagra Gour. Web Maker is a free and open-source project. To keep me motivated for working on such open-source and free side projects, I have launched a Patreon campaign.... -
4
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Katana is an open source command-line web crawling and spidering framework developed by ProjectDiscovery. It is designed to efficiently crawl websites and web applications in order to discover endpoints, resources, and other useful information that may not be easily visible through manual browsing. Katana focuses on speed and automation, making it suitable for use in security reconnaissance workflows and automated pipelines. Katana supports both standard HTTP crawling and headless browser... -
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
5
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
6
web-extension-starter
Web Extension starter to build "Write Once Run on Any Browser"
Web Extension starter to build "Write Once Run on Any Browser" extension. Cross Browser Support (Web-Extensions API). Browser Tailored Manifest generation. Automatic build on code changes. Auto packs browser-specific build files. SASS styling. TypeScript by default. ES6 modules support. React UI Library by default. Smart reload. Used by extensions in production that has over 100,000+ users. -
7
IPFS Web UI
A frontend for an IPFS node
A web interface to IPFS, shipped with Kubo, and ipfs-desktop. Check on your node stats, explore the IPLD powered merkle forest, see peers around the world and manage your files, without needing to touch the CLI. When working on the code, run an ipfs daemon, the local dev server, the unit tests, and the storybook component viewer and see the results of your changes as you save files. If you need to run IPFS in a Docker container, you can still have Web UI available by exposing both the... -
8
fess
Open source enterprise search server for websites, files, and data
...Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. It supports indexing and searching across many document formats including office documents, PDFs, and compressed archives. It also provides a web-based administrative interface that allows administrators to configure crawling targets, manage indexing tasks, and adjust search settings from a graphical dashboard. -
9
Playwright Skill for Claude Code
Claude Code Skill for browser automation with Playwright
...The system supports a wide range of use cases, including testing web applications, validating user interfaces, automating workflows, and extracting data from websites. One of its key advantages is its ability to generate custom Playwright code tailored to each request, allowing flexible and context-aware automation. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
Agregore Browser
A minimal browser for the distributed web (Desktop version)
A minimal web browser for the distributed web. Web Extension support. Built-in Markdown/Gemini/JSON rendering extension. Built-in QR code scanner and generator extension. Generate a QR code for the current page. Scan a QR code from the browser action window. Right-click a link or image to generate a QR code for it. Built-in ad blocker (ublock origin). Built-in support for creating web archives via ArchiveWeb.page. -
11
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across... -
12
Stylus for Chrome
Stylus - Userstyles Manager
Stylus is a fork of Stylish for Chrome, also compatible with Firefox as a WebExtension. "Stylus" is a fork of the popular Stylish extension which can be used to restyle the web. Not "ish", but "us", as in "us" the actual users. Stylus is a fork of Stylish that is based on the source code of version 1.5.2, which was the most up-to-date version before the original developer stopped working on the project. The objective in creating Stylus was to remove any and all analytics, and return to a more user-friendly UI. ... -
13
MetaMask
Mobile web browser providing access to websites that use the Ethereum
MetaMask is a mobile wallet that provides easy access to websites that use the Ethereum blockchain. A crypto wallet & gateway to blockchain apps. Start exploring blockchain applications in seconds. Trusted by over 30 million users worldwide. Available as a browser extension and as a mobile app, MetaMask equips you with a key vault, secure login, token wallet, and token exchange—everything you need to manage your digital assets. MetaMask provides the simplest yet most secure way to connect to... -
14
Ghostery
Ghostery Browser Extension for Firefox, Chrome, Opera and Edge
Ghostery helps you browse smarter by giving you control over ads and tracking technologies to speed up page loads, eliminate clutter, and protect your data. This is the unified code repository for the Ghostery browser extensions in Chrome, Firefox, Opera and Edge. Browse the web safer, faster & with less annoying ads. Equipped with award-winning AI anti-tracking technology to browse the websafe and quickly. Ghostery helps you stay informed about what companies are tracking you by listing the... -
15
ego (lite)
The fastest browser for AI agents to run web automation
Ego Lite is a macOS browser designed for people and external AI agents to work in parallel. Each agent receives an isolated Space, allowing several browser tasks to run simultaneously without taking over the user's active tabs. Optional Chrome data migration carries over logins, cookies, extensions, and bookmarks so agents can use authenticated sessions with less setup. The ego-browser skill connects tools such as Codex, Claude Code, Cursor, and custom agent CLIs to the browser. Agents... -
16
Return YouTube Dislike
Chrome extension to return youtube dislikes
Return YouTube Dislike is an open-source extension that returns the YouTube dislike count. Available for Chrome and Firefox as a Web Extension. Also available for other browsers as JS Userscript. Additionally, the dislike field in the YouTube API was removed on December 13th, 2021, removing any ability to judge the quality of content before watching. With the removal of dislike stats from the YouTube API, our backend switched to using a combination of scraped dislike stats, and estimates extrapolated from extension user data. ... -
17
Decap
A Git-based CMS for Static Site Generators
Open source content management for your Git workflow. Use Decap CMS with any static site generator for a faster and more flexible web project. Get the speed, security, and scalability of a static site, while still providing a convenient editing interface for content. Content is stored in your Git repository alongside your code for easier versioning, multi-channel publishing, and the option to handle content updates directly in Git. -
18
Betterfox
Firefox user.js for optimal privacy and security
Betterfox is an opinionated configuration profile for Mozilla Firefox designed to improve everyday web browsing by making the browser faster, more private, and more secure without relying on external add-ons or third-party code. Rather than being a separate browser, it consists of a curated set of preference tweaks (user.js settings) that users apply to their Firefox profile to optimize performance and harden privacy settings. The project focuses on a minimal-impact, maximum-effect approach,... -
19
Serverless Express by Vendia
Run Node.js web applications and APIs using existing frameworks
Run REST APIs and other web applications using your existing Node.js application framework (Express, Koa, Hapi, Sails, etc.), on top of AWS Lambda and Amazon API Gateway. Vendia is the real-time data cloud for rapidly building applications that securely share data across departments, companies, clouds, and regions. We’re excited to announce the latest release of Vendia Share! This release includes new features like smart contracts, user-level transactions, the beta release of Azure support,... -
20
JavaScript QR Code
Make QR Codes in JavaScript
A mini javascript library to get the famous QR code. You can set the size, content to be encoded and get the html code to include the image in your web pages. You can also share the image via email with a simple interface included in the mini library Now is available Desktop app, QR Code Generator, where Window 7 and later are supported. Both x86 and amd64 (x64) binaries are provided for Windows. Please note, the ARM version of Windows is not supported for now. Get Android app:... -
21
Middy
The stylish Node.js middleware engine for AWS Lambda
Middy is a very simple middleware engine that allows you to simplify your AWS Lambda code when using Node.js. If you have used web frameworks like Express, then you will be familiar with the concepts adopted in Middy and you will be able to get started very quickly. A middleware engine allows you to focus on the strict business logic of your Lambda and then attach additional common elements like authentication, authorization, validation, serialization, etc. in a modular and reusable way by... -
22
rebroswer-patches
Patches for Puppeteer and Playwright to reduce automation detection
rebrowser-patches is an open source collection of patches designed to improve the stealth capabilities of browser automation frameworks. It focuses primarily on enhancing Puppeteer and Playwright by modifying parts of their source code that may reveal automation activity to websites. Many modern websites rely on bot detection mechanisms that identify automation through behavioral or technical signals, and these patches aim to reduce those detection vectors. By applying targeted fixes, the... -
23
Loki
Visual Regression Testing for Storybook
There are a few visual regression tools for the web, but most either cannot be run headless or use phantomjs which is deprecated and a browser nobody is actually using. They usually also require you to maintain fixtures. With react-native it's now possible to target multiple platforms with a single code base, but there's no single tool to test all to my knowledge. Loki aims to have easy setup, no to low maintenance cost, reproducible tests independent of which OS they are run on, runnable on... -
24
Netlify CMS
A Git-based CMS for static site generators
Open source content management for your Git workflow. Use Netlify CMS with any static site generator for a faster and more flexible web project. Get the speed, security, and scalability of a static site, while still providing a convenient editing interface for content. Content is stored in your Git repository alongside your code for easier versioning, multi-channel publishing, and the option to handle content updates directly in Git. -
25
Jimp
An image processing library written entirely in JavaScript for Node
An image processing library for Node written entirely in JavaScript, with zero native dependencies. If you're using this library with TypeScript the method of importing slightly differs from JavaScript. Instead of using require, you must import it with ES6 default import scheme. If you're using a web bundles (webpack, rollup, parcel) you can benefit from using the module build of jimp. Using the module build will allow your bundler to understand your code better and exclude things you aren't...
