Search Results for "requests" - Page 2
Sort By:
Showing 58 open source projects for "requests"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
scrapy-proxy-headers
Handle custom proxy headers when making HTTPS requests in scrapy
The scrapy-proxy-headers package is designed for adding proxy headers to HTTPS requests in Scrapy. In normal usage, custom headers put in request.headers cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at Proxy Server Requests over HTTPS. Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at ProxyMesh made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. ... -
2
python-proxy-headers
Handle custom proxy headers when making HTTPS requests in python
The python-proxy-headers package provides support for handling custom proxy headers when making HTTPS requests in various python modules. We currently provide extensions to the following packages: urllib3 requests aiohttp httpx None of these modules provide good support for parsing custom response headers from proxy servers. And some of them make it hard to send custom headers to proxy servers. So we at ProxyMesh made these extension modules to support our customers that use Python and want to use custom headers to control our proxy behavior. ... -
3
ipfetch-python
Get your public IP Address on the Command Line
This small but fine Python Snippet requests the public IP adress (supports IPv4 and IPv6) at myip.is and outputs the result. It might be useful to get your public IP adress on the command line, or even to check whether your internet connection is working. USAGE : python ipfetch.py -
4
Web Link Collector 1000
Automatically collect all links from websites to a clean txt file
...- Smart Filtering: Include only same-domain links or gather external links too - Duplicate Prevention: Automatically removes duplicate links - Website-Friendly: Uses respectful delays between requests - Custom File Naming: Save your collections with custom meaningful names - Modern Interface: Clean design with status updates - Link Normalization: Standardizes URLs for proper formatting -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
5
spider_collection
Collection of Python web scraping scripts for data extraction tasks
spider_collection is a collection of Python web crawler scripts created primarily for experimentation, learning, and practical scraping tasks. spider_collection gathers multiple independent spiders designed to collect data from different platforms and services, demonstrating a variety of scraping techniques and workflows. These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages. Several scripts also incorporate multi-threading and proxy usage to improve scraping efficiency and help avoid common anti-scraping limitations. In addition to raw data collection, some spiders include basic data processing and analysis using tools such as pandas and simple visualization with matplotlib. ... -
6
autocrawler
Multiprocess Selenium crawler for downloading images by keywords
AutoCrawler is a Python-based image crawling tool designed to automatically download large numbers of images from search engines using automated browser interaction. It uses Selenium and a Chrome browser driver to navigate image search pages and collect image sources based on keywords provided by the user. AutoCrawler supports multiprocess and multithreaded downloading, which allows it to retrieve images faster by running several tasks simultaneously. Users provide search terms through a... -
7
PyNuker
A stress testing tool written in python.
...It infinitely(until stopped) sends a string of text via a UDP packet to a target computer or network device in an effort to flood the target with so much useless traffic that it stops responding to valid requests. I digitally sign some files in my releases. If you'd like to verify those signatures, you can find my PGP/GPG keys at: https://marcusadams.me/keys.html If you'd like to donate there's several ways to do so: PayPal: https://paypal.me/gerowen Bitcoin (BTC): bc1q86c5j7wvf6cw78tf8x3szxy5gnxg4gj8mw4sy2 Monero (XMR): 42ho3m9tJsobZwQDsFTk92ENdWAYk2zL8Qp42m7pKmfWE7jzei7Fwrs87MMXUTCVifjZZiStt3E7c5tmYa9qNxAf3MbY7rD LiberaPay: https://liberapay.com/gerowen -
8
dirhunt
Web crawler that finds hidden web directories without brute force
...Dirhunt processes HTML pages and other available sources to discover additional paths and directories while minimizing the number of requests sent to the server, making scans faster and less intrusive. It supports scanning multiple targets at the same time and allows results to be filtered, analyzed, and exported for further review. -
9
Scrapyd
A service daemon to run Scrapy spiders
...The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd is an application (typically run as a daemon) that listens to requests for spiders to run and spawns a process for each one. Scrapyd also runs multiple processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load. -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
10
DecryptLogin
Python library providing APIs for automated website login workflows
DecryptLogin is a Python library designed to simplify automated login processes for many popular websites by providing ready-to-use APIs that simulate authentication behavior. It focuses on implementing login mechanisms through HTTP requests, allowing developers to programmatically authenticate with supported services without manually replicating complex login flows. It includes modules that handle different authentication modes such as PC login, mobile login, and QR code login depending on what the target platform supports. DecryptLogin supports a wide variety of online services and platforms, including social media sites, developer platforms, cloud services, and other web portals. ... -
11
AutoScraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
This project is made for automatic web scraping to make scraping easy. It gets a URL or the HTML content of a web page and a list of sample data that we want to scrape from that page. This data can be text, URL or any HTML tag value of that page. It learns the scraping rules and returns similar elements. Then you can use this learned object with new URLs to get similar content or the exact same element of those new pages. -
12
pspider
Simple Python framework for building multithreaded web crawlers
PSpider is a lightweight web crawling framework written in Python designed to simplify the development of custom web spiders. It focuses on providing an easy-to-understand architecture while still supporting concurrent crawling for improved performance. It uses a multithreaded model that separates the crawling workflow into several components responsible for fetching, parsing, and saving data. Tasks are managed through queues, allowing different parts of the crawler to process work... -
13
Scylla
Intelligent proxy pool for collecting and managing public proxies
Scylla is an open source proxy pool system designed to collect, validate, and manage large numbers of public proxy servers for use in web scraping and data extraction workflows. It automatically crawls the internet to discover proxy IP addresses and evaluates their availability and reliability before adding them to a usable pool. It includes a JSON API that allows developers and applications to retrieve proxy information programmatically, making it easier to integrate proxy rotation into... -
14
instagram-profilecrawl
Instagram profile crawler that extracts posts, tags, and stats
...It crawls profile data such as follower counts, post information, hashtags, and other engagement-related metadata. It operates by automating a web browser using Selenium and performing requests to gather structured information from the platform. instagram-profilecrawl can analyze multiple usernames in a single run and store the extracted information locally in structured formats such as JSON. The collected data can include profile metadata, post details, engagement metrics, and commenter activity, allowing users to analyze account behavior or monitor profile growth over time. ... -
15
lxspider
Educational Python web scraping case collection for many sites
...It gathers numerous crawler implementations that demonstrate how to collect data from a wide range of websites and online services. It focuses heavily on practical cases that illustrate how different platforms handle requests, authentication parameters, and anti-scraping protections. lxSpider includes examples targeting areas such as e-commerce platforms, social media services, content sites, research databases, and information portals. Many of the cases explore techniques related to request analysis, signature generation, and reverse engineering that are often needed when interacting with modern web applications. ... -
16
ruia
Async Python framework for fast and flexible web scraping spiders
Ruia is an asynchronous web scraping micro-framework built for Python that focuses on simplicity, speed, and flexibility when creating web crawlers. Ruia is powered by Python’s asyncio library along with aiohttp, enabling developers to perform concurrent network requests efficiently and scrape data from websites with minimal overhead. Ruia follows a “write less, run faster” philosophy, emphasizing concise code and streamlined spider development. It provides a structured approach to building scraping projects through components such as data items, spiders, middleware, and plugins. Developers can define structured fields to extract information from HTML content and process responses asynchronously to improve crawling performance. ... -
17
GoogleScraper
Python tool for scraping search engine results from many providers
...GoogleScraper supports several major search engines and can be used to gather structured datasets from search result pages for further analysis. It provides two different scraping approaches: sending direct HTTP requests that simulate browser traffic or controlling real browsers through automation frameworks. By running automated queries and collecting results in bulk, the project can assist with tasks such as SEO research, trend discovery, or building datasets of websites related to specific keywords. GoogleScraper also includes capabilities for running multiple scraping tasks concurrently to improve performance and increase the amount of collected data. -
18
Requests-HTML
Pythonic HTML Parsing for Humans
...XPath Selectors, for the faint of heart. Mocked user-agent (like a real web browser). Automatic following of redirects. Connection–pooling and cookie persistence. The Requests experience you know and love, with magical parsing abilities, and async support. The rest of the code operates the same way as the synchronous version except that results is a list containing multiple response objects however the same basic processes can be applied as above to extract the data you want. -
19
ECommerceCrawlers
Collection of Python ecommerce and website crawler examples projects
ECommerceCrawlers is a collection of practical Python web crawler projects designed to gather data from a variety of ecommerce platforms, websites, and online services. It aggregates many independent crawler examples created by contributors and organized into separate subprojects that target specific sites or data sources. These examples demonstrate how to build and operate web scrapers capable of collecting structured information such as product listings, news content, job postings, social... -
20
ProxyBroker
Asynchronous tool for finding and checking public proxy servers
...ProxyBroker can filter proxies based on criteria such as anonymity level, response time, country of origin, and DNS blacklist status. In addition to discovering and validating proxies, it can also function as a proxy server that distributes incoming requests across a rotating pool of working proxies. This capability allows users to route traffic through multiple proxies automatically, which can help with tasks. -
21
WeChatSogou
Python library to crawl and retrieve data from WeChat accounts
...These components work together to process HTTP requests, handle verification mechanisms, and transform HTML or JSON responses into usable objects. Developers can integrate the library into scripts or larger data collection systems to automate gathering content from public accounts. -
22
ImageFap Gallery Downloader
Minimalistic ImageFap and xHamster gallery downloader (Python)
Note: the work on this project is currently on halt. Check out https://sourceforge.net/projects/imagefap-gallery-dl-cli/?source=directory for a more recent version. ImageFap Gallery Downloader is a Python script for full image gallery downloads on ImageFap and xHamster. xHamster support added in v0.3 In order to download user folders in v0.4, open the user profile, follow the 'Galleries' link and copy the folder link from the sidebar to the clipboard. Download everything from a user... -
23
That project aims at providing a clean API, and the corresponding C++ implementation, for parsing travel-focused requests (e.g., "washington dc beijing monday r/t +aa -ua 1 week 2 adults 1 dog").
-
24
Spondulas
Spondulas is browser emulator designed to retrieve web pages for hunti
Spondulas is browser emulator and parser designed to retrieve web pages for hunting malware. It supports generation of browser user agents, GET/POST requests, and SOCKS5 proxy. It can be used to parse HTML files sent via e-mail. Monitor mode allows a website to be monitored at intervals to discover changes in DNS or content over time. Autolog mode creates an investigation file that documents redirection chains. The retrieved web pages are parsed for links and reported to an output file. ... -
25
Command line script written in Python. Takes a screenshot of the entire desktop. Gives you the option to upload or not. If you choose to upload, requests server info. Then copies the code to the clipboard, so you're ready to share!
