Search Results for "web crawler source code"
Sort By:
Showing 30 open source projects for "web crawler source code"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
AI-powered service management for IT and enterprise teamsGive your IT, operations, and business teams the ability to deliver exceptional services—without the complexity. Maximize operational efficiency with refreshingly simple, AI-powered Freshservice.
-
1
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
2
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website.... -
3
eslint-plugin-compat
Check the browser compatibility of your code
Lint the browser compatibility of your code. Browser targets are configured using a browser list. You can configure browser targets in your package.json. If no configuration is found, browser list defaults to "> 0.5%, last 2 versions, Firefox ESR, not dead". Add polyfills to the settings section of your eslint config. Append the name of the object and the property if one exists. Toolchains for native platforms, like iOS and Android, have had API linting from the start. It's about time that... -
4
WireGuard Easy
The easiest way to run WireGuard VPN + Web-based Admin UI
WireGuard Easy is a streamlined solution for deploying and managing a WireGuard VPN server through a web-based interface, designed to eliminate the complexity typically associated with manual VPN configuration. It combines the WireGuard backend with a user-friendly admin dashboard, allowing users to control clients, monitor connections, and generate configuration files without interacting directly with command-line tools. The project is commonly deployed using Docker, making installation... -
Try Google Cloud Risk-Free With $300 in CreditUse your credit across every product. Compute, storage, AI, analytics. When it runs out, 20+ products stay free. You only pay when you choose to.
-
5
webiny
Enterprise open-source serverless CMS
Enterprise open-source serverless CMS. Includes a headless CMS, page builder, form builder and file manager. Easy to customize and expand. Deploys to AWS. Create GraphQL APIs, full-stack applications and websites. Deploy with single command to your AWS. Runs on services like AWS Lambda and DynamoDB. Highly-scalable & highly-available out of the box. You get a full-stack project with a GraphQL API and a React frontend that you can use to start building. Write custom apps and business logic... -
6
Proton VPN Browser Extension
Proton VPN Browser Extension
The Proton VPN Browser Extension repository houses the code for an official browser extension that lets users quickly secure their web browsing traffic through ProtonVPN from within browsers like Firefox and Chrome without routing all system traffic. This extension provides users with a convenient way to encrypt and anonymize HTTP requests, hide IP addresses, and prevent tracking while browsing, acting independently of the full OS-level VPN clients. Because browser extensions are constrained... -
7
crawlee
A web scraping and browser automation library for Node.js
Crawlee is a web scraping and browser automation library. It helps you build reliable crawlers. Fast. Crawlee won't fix broken selectors for you (yet), but it helps you build and maintain your crawlers faster. When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back. It keeps your proxies healthy by rotating them smartly with good fingerprints that... -
8
Mini QR
Create & scan cute qr codes easily
Mini QR is a web app focused on making QR codes feel friendly and design-forward, combining a polished QR generator with a built-in scanner so you can both create and decode codes in the same place. It emphasizes customization so the QR you generate can match a brand, event theme, or personal style, including color and styling controls, framed layouts with labels, and the ability to add a logo image. Because QR reliability matters as much as looks, it exposes practical settings like error... -
9
FlareSolverr
Proxy server to bypass Cloudflare protection
FlareSolverr is a proxy server to bypass Cloudflare and DDoS-GUARD protection. FlareSolverr starts a proxy server, and it waits for user requests in an idle state using few resources. When some request arrives, it uses puppeteer with the stealth plugin to create a headless browser (Firefox). It opens the URL with user parameters and waits until the Cloudflare challenge is solved (or timeout). The HTML code and the cookies are sent back to the user, and those cookies can be used to bypass... -
Earn up to 16% annual interest with Nexo.Put idle assets to work with competitive interest rates, borrow without selling, and trade with precision. All in one platform. Geographic restrictions, eligibility, and terms apply.
-
10
JSONView
A web extension that helps you view JSON documents in the browser
A web extension that helps you view JSON documents in the browser. Normally when encountering a JSON document (content type application/json), Firefox simply prompts you to download the view. With the JSONView extension, JSON documents are shown in the browser similar to how XML documents are shown. The document is formatted, highlighted, and arrays and objects can be collapsed. Even if the JSON document contains errors, JSONView will still show the raw text. JSONView is a Web extension... -
11
CapRover
Scalable PaaS (automated Docker+nginx), aka Heroku on Steroids
CapRover is an extremely easy-to-use app/database deployment & web server manager for your NodeJS, Python, PHP, ASP.NET, Ruby, MySQL, MongoDB, Postgres, WordPress (and etc...) applications! It's blazingly fast and very robust as it uses Docker, Nginx, LetsEncrypt and NetData under the hood behind its simple-to-use interface. For a developer who does not like spending hours and days setting up a server, building tools, sending code to the server, building it, getting an SSL certificate,... -
12
MemFree
Hybrid AI Search Engine & AI Page Generator
memfree is an open source hybrid AI search engine and page generation platform designed to help users retrieve information from both personal knowledge bases and the public web through a unified interface. The project combines retrieval-augmented search with AI summarization to deliver concise answers instead of forcing users to manually sift through multiple sources. -
13
Dev Browser
A Claude Skill to give your agent the ability to use a web browser
Dev Browser is a browser automation skill/plugin that enables an AI agent to control a real browser for verification and testing during development. Its purpose is to close the gap between “code was written” and “the UI actually works,” by letting the agent navigate, interact with pages, and validate behavior in a live environment. A key idea is persistence: the browser can keep pages open so the agent can navigate once and then perform multiple interactions across scripts without losing... -
14
Routr
The future of programmable SIP servers
Lightweight sip proxy, location server, and registrar that provides a reliable and scalable SIP infrastructure for telephony carriers, communication service providers, and integrators. Routr provides all the tools required to deploy your VoIP network, including a command-line for remote server control. It can also be controlled via a RESTful API or a web-based GUI. Routr includes all tools for deploying your VoIP network. It offers remote server control via command-line, RESTful API, or a... -
15
AWS Lambda NodeJS Runtime Interface
Extend your preferred base images to be Lambda compatible
We have open-sourced a set of software packages, Runtime Interface Clients (RIC), that implement the Lambda Runtime API, allowing you to seamlessly extend your preferred base images to be Lambda compatible. The Lambda Runtime Interface Client is a lightweight interface that allows your runtime to receive requests from and send requests to the Lambda service. The Lambda NodeJS Runtime Interface Client is vended through npm. You can include this package in your preferred base image to make... -
16
Proton Drive SDK
SDK repository intended to provide reusable client libraries
Proton Drive SDK is Proton’s dedicated SDK repository intended to provide reusable client libraries for integrating Proton Drive capabilities into other applications, with an initial focus on a JavaScript SDK and plans for a C# SDK. The repository is explicitly labeled as a work in progress and notes that it is not ready for use yet, which signals that public APIs and stability guarantees should not be assumed at this stage. Its implementation is predominantly TypeScript, which is... -
17
Brill Software
A faster way to develop React Web Applications
The Brill Framework allows React web applications to be built quickly using a "Low Code" approach. A Content Management System (CMS) supports editing of pages containing React components. The React components communicate with each other and the Server using a middleware that's based on WebSockets. With a "No Code" solution, there's always something you require that's not support. You spend ages bending the product to your requirements or pay the supplier to provide the components... -
18
AeroFTP
AeroFTP is a Cross-platform desktop client for FTP, SFTP, WebDAV, S3
AeroFTP is a cross-platform file transfer client that goes beyond traditional FTP. Connect to 25+ protocols, FTP/FTPS, SFTP, WebDAV, S3, Google Drive, Dropbox, OneDrive, MEGA, Box, pCloud, Azure, Filen, and more from a single interface. Security-first: AeroVault v2 encrypted containers (AES-256-GCM-SIV), Cryptomator support, and zero telemetry. Built-in AeroAgent AI assistant with 19 providers and 47 tools for file operations and workflow automation. Includes Monaco editor,... -
19
...You can also connect directly with the companies supporting the development. Source code available on github https://github.com/geonetwork/
-
20
Plasmic
Build apps, websites, and content. Integrate with your codebase
Plasmic is a visual development platform designed to help teams design, build, and manage websites and web applications through a combination of visual editing tools and code integration. The platform provides a drag-and-drop design environment that allows designers, developers, and product teams to collaborate on building user interfaces while maintaining compatibility with modern front-end frameworks. Instead of requiring developers to manually code every layout, Plasmic allows visual... -
21
Ayakashi
The next generation web scraping framework
The next-generation web scraping framework. The web has changed. Gone are the days when raw HTML parsing scripts were the proper tool for the job. Javascript and single-page applications are now the norms. Demand for data scraping and automation is higher than ever, from business needs to data science and machine learning. Our tools need to evolve. Ayakashi helps you build scraping and automation systems that are easy to build simple or sophisticated, highly performant, maintainable, and... -
22
Outline Client
Outline clients, developed by Jigsaw
Outline clients, developed by Jigsaw. The Outline clients use the popular Shadowsocks protocol, and lean on the Cordova and Electron frameworks to support Windows, Android / ChromeOS, Linux, iOS and macOS. The Outline Client is a cross-platform VPN or proxy client for Windows, macOS, iOS, Android, and ChromeOS. The Outline Client is designed for use with the Outline Server software, but it is fully compatible with any Shadowsocks server. The client's user interface is implemented in Polymer... -
23
Outline Apps
Outline Client and Manager, developed by Jigsaw
Codecov currently only tracks the /client Apple libraries and core web view code. Outline makes it easy for anyone to create a VPN server, allowing you to share access to the free and open internet with those in need. A cross-platform proxy client for Windows, macOS, iOS, Android, and Linux. The Outline Client is designed for use with the server deployed with the Outline Manager, but it is also fully compatible with any Shadowsocks server. -
24
generator-ngx-rocket
Extensible Angular 13+ enterprise-grade project generator
Extensible Angular 5+ enterprise-grade project generator based on angular-cli with best practices from the community. Bootstrap, Angular Material or even Ionic are available, just pick one and start to code. Save time and maintenance effort by creating Progressive Web App and mobile app with the same code base. The app template is based on HTML5, TypeScript and Sass. Optimized build and bundling process with Webpack. Development server with backend proxy and live reload. Cross-browser CSS... -
25
Dot Browser for Android
Dot Browser for Android
...We will offer to mask your email address when you sign up for sites or services. Dot is built on top of open-source software meaning the source code is open to anyone. Dot can migrate all your data from your previous browser in just a few clicks. Dot Browser has Dot Shield, which protects you from nasty ads and trackers. We chose Gecko due to its position as the best competitor to Chromium, and because it’s developed by Mozilla, which are best known for their privacy efforts in the community.
