Search Results for "python web crawler"
Sort By:
Showing 15 open source projects for "python web crawler"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
dirsearch
Web path scanner
An advanced command-line tool designed to brute force directories and files in webservers, AKA web path scanner. Wordlist is a text file, each line is a path. About extensions, unlike other tools, dirsearch only replaces the %EXT% keyword with extensions from -e flag. For wordlists without %EXT% (like SecLists), -f | --force-extensions switch is required to append extensions to every word in wordlist, as well as the /. To use multiple wordlists, you can separate your wordlists with commas.... -
2
SiteOne Crawler (desktop app)
A free, feature-rich web analyzer and exporter/cloner you will love!
A free in-depth website analyzer providing audits of security, performance, SEO, accessibility and other technical aspects. Available as a desktop application for Windows/macOS/Linux and as a CLI tool for advanced users and CI/CD processes. It also includes an offline web page exporter (website clone, mirror). -
3
Kaggle CLI
The official CLI to interact with Kaggle
Kaggle CLI is Kaggle’s official command-line interface for interacting with the Kaggle platform from a terminal. It lets users authenticate, search resources, download files, submit competition entries, manage datasets, work with models, run notebooks, and read discussion content without relying only on the web interface. The tool is useful for data scientists who want to automate Kaggle workflows inside scripts, CI jobs, notebooks, or reproducible local environments. It supports both... -
4
Claude Engineer v3
Claude Engineer is an interactive command-line interface (CLI)
Claude Engineer v3 is a self-improving AI software development assistant built around Claude 3.5 Sonnet. It provides both a command-line interface and a modern web interface for coding-related workflows. The project is designed to let Claude identify missing capabilities, generate new tools, load them dynamically, and use them in later conversations. It includes real-time token tracking, conversation state management, tool usage indicators, and structured configuration options. The web... -
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
5
aws-cli
Universal Command Line Interface for Amazon Web Services
The AWS CLI is the universal command-line interface for managing AWS services, automating tasks, and scripting cloud workflows. It exposes nearly every public API from EC2 and S3 to IAM, Lambda, and beyond, providing parity with the service SDKs in a tool you can run anywhere. Profiles, regions, single-sign-on, and credential helpers make it straightforward to switch contexts securely across accounts and environments. Its output controls and JMESPath querying let you slice, filter, and... -
6
HTTPie CLI
Modern, user-friendly command-line HTTP client for the API era
HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers. The HTTP & HTTPS commands allow for creating and sending arbitrary HTTP requests. They use simple and natural syntax and provide formatted and colorized output. -
7
cheat.sh
The only cheat sheet you need
cheat.sh is a compact, network-accessible cheat-sheet service that serves concise examples and usage notes for hundreds of shell commands, programming languages, and tools via a simple HTTP interface. You can query it from the terminal (for example curl cht.sh/rsync or curl cheat.sh/ls) or browse the web front page; it also supports a shorthand hostname (cht.sh) and provides both online and standalone/local installation modes. The repository contains the server and client code, instructions to run a local standalone instance (including Python virtualenv setup), and tooling to fetch or maintain the upstream cheat-sheet data; installation documentation explains disk-space needs and dependency setup for offline use. ... -
8
shot-scraper
A command-line utility for taking automated screenshots of websites
shot-scraper is a command-line utility for taking automated screenshots of web pages using a headless browser engine. After installation, a single command can capture a full-page screenshot of a URL and save it to a file, making it ideal for documentation, monitoring, and visual regression tasks. Under the hood it uses a modern browser (installed via a one-time shot-scraper install step) and exposes options for viewport size, full-page versus clipped screenshots, and device emulation. Beyond... -
9
CineCLI
CineCLI is a cross-platform command-line movie browser
CineCLI is a command-line utility designed to help movie lovers quickly browse, search, and access film information from the terminal without needing a graphical interface. It connects to popular online movie databases to fetch metadata such as titles, release dates, ratings, genres, casts, posters, and plot summaries, presenting all of that in a concise, text-friendly format suitable for terminals or scripts. Users can search by keyword, year, or exact title and then drill into detailed... -
Veeam Data Platform v13.1Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
10
s3cmd
Command line tool for managing Amazon S3 and CloudFront services
Open-source tool to access Amazon S3 file storage. S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage. Lots of features and options have been added to s3cmd since its very first release in 2008.... we recently counted more than 60 command line options, including multipart uploads, encryption, incremental backup, s3 sync, ACL and Metadata... -
11
Docksal
Docksal is a tool for defining and managing web development
Running Docker on macOS and Windows is supported via VirtualBox as well as Docker for Mac / Docker for Windows. Docksal project stacks are pre-loaded with common development tools, which include Composer, PHP Code Sniffer, php-cli, node, npm, ruby, bundler, and python. For PHP, Drupal, and WordPress development, it comes with Drush, Drupal Console, and WP-CLI. There is built-in support for Apache Solr, Varnish, Memcache, Selenium, and Behat. Since services are containerized with... -
12
SIGAR (System Information Gatherer and Reporter) is a cross-platform, cross-language library and command-line tool for accessing operating system and hardware level information in Java, Perl and .NET.
-
13
Nanon is a web page manager, ideal for simple sites with no server generated content. It lets you define templates and reusable html code blocks like menus and ad sections. Nanon is a spiffy command line tool that you can learn how to use in minutes.
-
14
PyUserCreate is a tool for *NIX OSes that allows you to accept user account requests from a web interface, and use a simple command-line tool to approve/disprove requests.
-
15
Reddo is a command line tool to translate any pair of languages using web translators such google, babelfish or others.
- Previous
- You're on page 1
- Next
