Search Results for "web crawler source code"
Sort By:
100 projects for "web crawler source code" with 2 filters applied:
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
3
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
4
claude-code-transcripts
Tools for publishing transcripts for Claude Code sessions
claude-code-transcripts is a command-line utility that takes session files exported from Claude Code (in JSON or JSONL format) and turns them into clean, navigable HTML transcripts that can be viewed in any modern web browser. It is designed to make the often dense and verbose outputs from AI coding sessions easier to read, share, and archive by breaking conversations into paginated, annotated pages with navigable timelines of prompts and responses. Users can run this tool locally or fetch... -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
5
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
6
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
7
GH Archive
GH Archive is a project to record the public GitHub timeline
...The dataset is also published through Google BigQuery for large-scale SQL-style exploration without downloading every archive. Its structure supports trend analysis, visualizations, machine learning, and open-source ecosystem research. The repository contains the crawler, supporting scripts, and website code, while the actual event files are hosted separately. -
8
Sycamore
A library for creating reactive web apps in Rust and WebAssembly
A reactive library for creating web apps in Rust and WebAssembly. Sycamore harnesses the full power of Rust via WebAssembly, giving you full control over performance. Write code that feels natural. Everything is built on reactive primitives without a cumbersome virtual DOM. No JavaScript. Had enough of JavaScript? So have we. Create apps using Sycamore without touching a single line of JS. -
9
Coraza
OWASP Coraza WAF is a golang modsecurity compatible firewall library
...Coraza runs the OWASP Core Rule Set (CRS) to protect your web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. CRS protects from many common attack categories including: SQL Injection (SQLi), Cross Site Scripting (XSS), PHP & Java Code Injection, HTTPoxy, Shellshock, Scripting/Scanner/Bot Detection & Metadata & Error Leakages. Coraza is a library at its core, with many integrations to deploy on-premise Web Application Firewall instances. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
Prism
Lightweight, robust, elegant syntax highlighting
Prism is a lightweight, extensible syntax highlighter, built with modern web standards in mind. It’s used in millions of websites, including some of those you visit daily. -
11
Testcontainers node
Supports tests, providing lightweight, throwaway instances of database
Testcontainers is an open source library for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container. No more need for mocks or complicated environment configurations. Define your test dependencies as code, then simply run your tests and containers will be created and then deleted. -
12
Tamagui
Style React fast with 100% parity on React Native
Tamagui is a cross-platform UI framework and styling system for React and React Native that enables developers to build high-performance applications with a single codebase shared across web and mobile platforms. It combines a universal style system with a powerful optimizing compiler that transforms styled components into efficient, platform-specific output, significantly improving runtime performance. The framework provides a superset of the React Native styling API, allowing developers to... -
13
Pannellum
Pannellum is a lightweight, free, and open source panorama viewer
Pannellum is a lightweight, free, and open source panorama viewer for the web. Built using HTML5, CSS3, JavaScript, and WebGL, it is plug-in free. It can be deployed easily as a single file, just 21kB gzipped, and then embedded into pages as an <iframe>. A configuration utility is included to generate the required code for embedding. An API is included for more advanced integrations. -
14
Hutool
A set of tools that keep Java sweet
Hutool is a small but comprehensive Java tool class library. Through static method encapsulation, it reduces the learning cost of related APIs, improves work efficiency, makes Java as elegant as a functional language, and makes the Java language "sweet". The tools and methods in Hutool come from each user's meticulous attention to detail. It covers all aspects of the underlying code of Java development. It is not only a sharp tool to solve small problems in large-scale project development,... -
15
COBE
5KB WebGL globe lib
The COBE project is a lightweight WebGL-based library for rendering interactive 3D globes in web applications. It is designed to provide visually appealing globe visualizations with minimal setup and high performance. The library focuses on simplicity, allowing developers to create animated and interactive globe experiences with just a few lines of code. It supports smooth animations, rotation, and customizable visual styles, making it suitable for dashboards, landing pages, and data... -
16
JsStore
A complete IndexedDB wrapper with SQL like syntax
Execute Database operation in browsers with JsStore. JsStore is an indexedDB wrapper. It makes IndexedDB super easy with its SQL-like APIs. Harness the power of JsStore to streamline database operations in your web applications. With its SQL-like API, JsStore simplifies IndexedDB interactions, enabling developers to easily query, filter, and manipulate data with familiar syntax and efficiency. -
17
multiOTP open source
PHP strong authentication library, web interface & CLI, OATH certified
multiOTP is a PHP class, a powerful command line utility and a web interface developed by SysCo systèmes de communication sa in order to provide a completely free and easy operating system independent server side implementation for strong two factors authentication solution. multiOTP supports hardware and software tokens with different One-Time Password algorithms like OATH/HOTP, OATH/TOTP and mOTP (Mobile-OTP). QRcode generation is also embedded in order to support provisioning of Google... -
18
Plotly Graphing Library for MATLAB
Plotly Graphing Library for MATLAB
Create interactive charts in your web browser with MATLAB® and Plotly. -
19
AtlantaFX
Modern JavaFX CSS theme collection with additional controls
Modern JavaFX CSS theme collection with additional controls. -
20
100 proyectos de JavaScript
100 JavaScript projects with code and videos for free
100 proyectos de JavaScript is a large collection of 100 JavaScript projects, each with source code and an accompanying video tutorial, offered entirely for free. The repository is tied to the website javascript100.dev, which presents the projects as a structured learning path so learners can progressively build their skills through practice.Each project lives in its own numbered folder (for example, 01-tinder-swipe), and the README explains how to run them via the VS Code Live Preview... -
21
Lyricist
Missing I18N/L10N (internationalization/localization) multiplatform
Jetpack Compose greatly improved the way we build UIs on Android, but not how we interact with strings. stringResource() works well, but doesn't benefit from the idiomatic Kotlin like Compose. Lyricist tries to make working with strings as powerful as building UIs with Compose, i.e., working with parameterized string is now typesafe, use of when expression to work with plurals with more flexibility, and even load/update the strings dynamically via an API. -
22
Helium
Lighter web automation with Python
Helium is a Python library built on top of Selenium to make browser automation more intuitive and human-friendly. It replaces verbose boilerplate code with natural language-like API calls such as click("Login") or write("hello", into="Name"). Helium manages browser setup, waits, and teardown, enabling quick development of scripts for testing, scraping, or task automation without requiring deep Selenium knowledge. -
23
Gaufrette
PHP library that provides a filesystem abstraction layer
Gaufrette is a PHP library designed to abstract filesystem operations, providing a consistent interface to interact with various storage backends. It allows developers to work with files stored locally, on cloud platforms, or via network protocols, without changing the code. Gaufrette supports plugins for caching, metadata management, and file transformation, making it a flexible choice for file management in web applications. -
24
Hello ImGui
Hello, Dear ImGui: unleash your creativity in app development
hello_imgui is a cross-platform framework that simplifies the development of graphical applications using Dear ImGui. It abstracts away boilerplate code for window creation, input handling, and platform integration, allowing developers to focus on building user interfaces and interactive content quickly. Targeted at prototyping, tools, and GUI apps, hello_imgui works on Windows, macOS, Linux, iOS, and WebAssembly with minimal setup. Its plug-and-play nature makes it ideal for teaching,... -
25
next-starter
A Next.js starter template, packed with features
next-starter is a comprehensive Next.js starter template designed to accelerate project initialization. It integrates essential tools and technologies, including TypeScript, Tailwind CSS, NextAuth.js, ESLint, Stripe, and various testing utilities, providing a robust foundation for building scalable and maintainable web applications.