Search Results for "web crawler source code"
Sort By:
Showing 58 open source projects for "web crawler source code"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
GH Archive
GH Archive is a project to record the public GitHub timeline
...The dataset is also published through Google BigQuery for large-scale SQL-style exploration without downloading every archive. Its structure supports trend analysis, visualizations, machine learning, and open-source ecosystem research. The repository contains the crawler, supporting scripts, and website code, while the actual event files are hosted separately. -
2
Ruby on Rails
Web Application Framework written in Ruby
Rails is a web application development framework written in the Ruby programming language. It is designed to make programming web applications easier by making assumptions about what every developer needs to get started. It allows you to write less code while accomplishing more than many other languages and frameworks. Experienced Rails developers also report that it makes web application development more fun. -
3
Password Pusher
An application to securely communicate passwords over the web
Give your users the tools to be secure by default. Password Pusher is an open source application to communicate passwords over the web. Links to passwords expire after a certain number of views and/or time has passed. Only enter a password into the box. Other identifying information can compromise security. All passwords are encrypted prior to storage and are available to only those with the secret link. Once expired, encrypted passwords are unequivocally deleted from the database. -
4
kimuraframework
AI-first Ruby framework for building fast, flexible web scraping spide
Kimurai is an open source web scraping framework written in Ruby that simplifies the process of building automated data extraction tools. It provides a clean domain-specific language that allows developers to define scraping logic and data schemas with minimal boilerplate code. Kimurai can use AI-assisted extraction to identify where data resides in HTML pages, automatically generating selectors that are cached for future use so subsequent scraping runs operate with pure Ruby performance. ... -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
gist(1)
Potentially the best command line gister
gist(1) is a web-based code and text sharing platform created to allow developers to quickly store, share, and collaborate on snippets of code or notes. It integrates closely with Git, enabling version control, revisions, and forking, which makes it suitable for both temporary sharing and long-term storage of reusable code fragments. Users can create public or private gists, depending on whether they want to share their content openly or keep it accessible only via a link. The platform... -
6
DevDocs
API Documentation Browser
The devdocs repository powers the DevDocs web application, a fast, offline-friendly documentation browser for many programming languages, libraries, and APIs. It aggregates documentation from multiple sources (e.g., MDN, Python, Ruby, Git, etc.), converts them into a uniform format, and indexes them for instant text searching. The codebase includes a backend that handles ingestion, parsing, and transformation of documentation sources into a static site structure, as well as the client side... -
7
wombat
Lightweight Ruby DSL for scraping structured data from web pages
Wombat is a lightweight web crawling and scraping library written in Ruby that focuses on extracting structured data from web pages using a concise domain-specific language (DSL). It is designed to simplify the process of defining how information should be collected from HTML documents without requiring large amounts of scraping boilerplate code. Developers can declare the data fields they want and specify selectors or rules for retrieving them, allowing Wombat to parse and return structured... -
8
Sinatra
Classy web-development dressed in a DSL
Sinatra is a minimalist Ruby web framework that focuses on simplicity and flexibility, letting developers define web applications in just a few lines of code. Its DSL maps routes directly to Ruby blocks, making it straightforward to build APIs, small services, and quick prototypes without the overhead of full-stack frameworks like Rails. Despite its simplicity, Sinatra supports middleware, templates, sessions, filters, and helpers, so it can grow with more complex requirements. The framework... -
9
Pro Git
Pro Git 2nd Edition
...Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later. For the examples in this book, you will use software source code as the files being version controlled, though in reality you can do this with nearly any type of file on a computer. If you are a graphic or web designer and want to keep every version of an image or layout (which you would most certainly want to), a Version Control System (VCS) is a very wise thing to use. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
jekyll-theme-8bit
A Jekyll theme inspired by classic 8bit games
A Jekyll theme inspired by classic 8-bit games. Jekyll-theme-8bit has been scaffolded by the Jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. Snippets of code within the _includes directory that can be inserted in multiple layouts (and another include-file as well) within the same theme-gem. -
11
Puma
A Ruby/Rack web server built for concurrency
Unlike other Ruby Webservers, Puma was built for speed and parallelism. Puma is a small library that provides a very fast and concurrent HTTP 1.1 server for Ruby web applications. It is designed for running Rack apps only. What makes Puma so fast is the careful use of a Ragel extension to provide fast, accurate HTTP 1.1 protocol parsing. This makes the server scream without too many portability issues. If you are using Bundler, just add Puma to your project's Gemfile. Once you've installed... -
12
Jazzy
Soulful docs for Swift & Objective-C
jazzy is a command-line utility that generates documentation for Swift or Objective-C. Instead of parsing your source files, jazzy hooks into Clang and SourceKit to use the AST representation of your code and its comments for more accurate results. The output matches the look and feel of Apple’s official reference documentation, post WWDC 2014. Jazzy can also generate documentation from compiled Swift modules using their symbol graph instead of source code. You need development tools to... -
13
DocuSeal
Open source DocuSign alternative
Open source document filling and signing. DocuSeal is an open-source platform that provides secure and efficient digital document signing and processing. Create PDF forms to have them filled and signed online on any device with an easy-to-use, mobile-optimized web tool. Use embeddable code snippets to seamlessly implement the document signing workflows directly on your website or app. -
14
ciao
HTTP checks & tests (private & public) monitoring
HTTP checks & tests (private & public) monitoring - check the status of your URL. ciao checks HTTP(S) URL endpoints for a HTTP status code (or errors on the lower TCP stack) and sends a notification on status change via E-Mail or Webhooks. It uses Cron syntax to schedule the checks and comes along with a Web UI and a RESTful JSON API. Create an open-source web application for checking URL statuses with a UI and a REST API which is easy to install and maintain (no external dependencies like Databases, Caches, etc.) in public and private environments. -
15
bootstrap_form
Repository of the bootstrap_form gem
Official repository of the bootstrap_form gem, a Rails form builder that makes it super easy to create beautiful-looking forms using Bootstrap 5. bootstrap_form is a Rails form builder that makes it super easy to integrate Bootstrap v5-style forms into your Rails application. It provides form helpers that augment the Rails form helpers. bootstrap_forms's form helpers generate the form field and its label and all the Bootstrap mark-up required for proper Bootstrap display. bootstrap_form. You... -
16
if-me.org
Free, open source mental health communication web app
if-me.org is a community for mental health experiences that encourages people to share their personal stories with trusted allies. Trusted allies are the people we interact with on a daily basis, including friends, family members, co-workers, teachers, and mental health workers. Dealing with mental health is what makes us human. But for a lot of us, it's a struggle to be open about it. Not everyone is a counselor or therapist. The people who we interact with every day shape our emotions and... -
17
StimulusReflex
Build reactive applications with the Rails tooling you already know
StimulusReflex is a Ruby on Rails framework for building reactive, real-time web interfaces without moving most application logic into a heavy frontend framework. It connects Stimulus controllers, Rails server-side actions, Action Cable, and CableReady to respond to user interactions over WebSockets. When a user clicks, types, submits, or triggers another event, the server processes the action and sends DOM updates back to the browser. This lets developers keep validation, persistence,... -
18
wappskafander_t2
Wraps a HTTP server that might optionally run PHP by using FastCGI.
wappskafander_t2 wraps an old version of a Hiawatha web server (hiawatha-webserver.org). If FastCGI and PHP are available, then the web server probably can execute PHP. As of 2022_11 this "branch" of the wappskafander_t2 will not be incrementally updated, because it, the wrapping code, NOT the wrapped web server, is an old mess that needs a total rewrite. The current version of the wrapping code is usable as a functional "blob" that serves HTTP and optionally PHP generated content from... -
19
Bridge Designer and Contest
A national virtual bridge engineering contest for kids of all ages.
This version of Bridge Designer is deprecated. Release distribution binaries have been removed since 1 July 2026. The new ASCE Bridge Designer is a web app available at http://asce.org/bridge-designer. Please use and enjoy the new app! We thank all project supporters and users for decades of contributions. The new support channel is bridgedesignerteam@gmail.com The national contest is no longer active. Several regional contests exist worldwide. The new Bridge Designer includes tools... -
20
Jekyll Pure Liquid Table of Contents
A GitHub Pages compatible Table of Contents generator
GitHub Pages can't run custom Jekyll plug-ins so when generating Tables of Contents (TOCs), you're stuck with either a JavaScript solution or using kramdown's {:toc} option. However, by using {:toc}, you are forced to have that code next to your actual markdown and you can't place it in a layout. This means every. single. post. will need to have the snippet. If you choose the JavaScript approach, that's perfectly fine but what if JS is disabled on someone's browser or your page is just... -
21
GitLab CE Server For Local Intranets
The Free & Popular Community git Server in a Complete Virtual Machine
This VM is created for 2 reasons: 1. Very little initial setup work required to make a GIT Server live, within minutes. 2. This system should keep running for Years, without requiring Updates / Breakages. If you are new to Virtual Machines, then please watch the Video below ( taken from my other project. just replace td with gi wherever mentioned ) After starting this VM, please login to its administration panel with: Website Address: https://gi.local/ ( Accept Any Warnings due to... -
22
Premonition
Add block-styled content to your site. Like summaries, notes and hints
Premonition is a Jekyll plugin that can transform Markdown blockquotes into styled blocks of code. The default template and stylesheet focus on creating info boxes, but through the templating system, you can modify it to suit your needs. With version 4 we also introduced a new citation box. -
23
Jekyll::Asciinema
Liquid tag for embedding asciicasts recorded with asciinema
Provides a Liquid tag for embedding asciicasts recorded with asciinema for use in Jekyll sites. After checking out the repo, run bin/setup to install dependencies. Then, run bin/console for an interactive prompt that will allow you to experiment. -
24
brutalist-blog
Minimalistic theme that automatically supports dark & light modes
Minimalistic Jekyll theme that automatically supports dark and light modes for blogs. -
25
Dato.rss
The best RSS Search experience you can find
dato.rss is a Ruby gem that provides a simple DSL for generating RSS 2.0 feeds. It allows developers to build XML-compliant RSS feeds directly within Ruby code using clean, readable syntax. Ideal for blogs, news aggregators, and content-heavy applications, dato.rss helps deliver structured content feeds without dealing with raw XML.
