Search Results for "web extract" - Page 2
Sort By:
Showing 50 open source projects for "web extract"
View related business solutions-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
Tarsier
Vision utilities for web interaction agents
At Reworkd, we iterated on all these problems across tens of thousands of real web tasks to build a powerful perception system for web agents... Tarsier! In the video below, we use Tarsier to provide webpage perception for a minimalistic GPT-4 LangChain web agent. Tarsier visually tags interactable elements on a page via brackets + an ID e.g. [23]. In doing this, we provide a mapping between elements and IDs for an LLM to take actions upon (e.g. CLICK [23]). We define interactable elements... -
2
fastdup
An unsupervised and free tool for image and video dataset analysis
fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale. -
3
spider_collection
Collection of Python web scraping scripts for data extraction tasks
spider_collection is a collection of Python web crawler scripts created primarily for experimentation, learning, and practical scraping tasks. spider_collection gathers multiple independent spiders designed to collect data from different platforms and services, demonstrating a variety of scraping techniques and workflows. These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages. ... -
4
It is a powerful data scraping tool that helps you extract information from various online sources. Easily collect data from Google SERP, Maps, Shopify, Zillow, and more. With a user-friendly interface, you can scrape and save data in JSON or Excel formats. Unlock insights from the web effortlessly with scrape-it.cloud API.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
mlscraper
ML-based HTML scraper that learns extraction rules from examples
mlscraper is a Python library designed to automatically extract structured data from HTML pages without requiring developers to manually write CSS selectors or XPath rules. Instead of defining extraction logic by hand, users provide a few examples of the data they want to retrieve from a webpage. It analyzes those examples within the HTML document and determines patterns or rules that can be used to extract the same type of information from similar pages. -
6
instagram-profilecrawl
Instagram profile crawler that extracts posts, tags, and stats
instagram-profilecrawl is a Python-based automation script designed to collect publicly available information from Instagram profiles. It crawls profile data such as follower counts, post information, hashtags, and other engagement-related metadata. It operates by automating a web browser using Selenium and performing requests to gather structured information from the platform. instagram-profilecrawl can analyze multiple usernames in a single run and store the extracted information locally... -
7
HostHunter
OSINT reconnaissance tool for discovering hostnames from IP addresses
HostHunter is an open source reconnaissance tool designed to discover and extract hostnames associated with a large set of IPv4 or IPv6 addresses. It helps security professionals map IP addresses to virtual hostnames using a combination of OSINT data sources and active reconnaissance techniques. This approach enables users to identify hidden or additional services that may be hosted behind a single IP address. By correlating hostname information from certificates, APIs, HTTP headers, and... -
8
ruia
Async Python framework for fast and flexible web scraping spiders
Ruia is an asynchronous web scraping micro-framework built for Python that focuses on simplicity, speed, and flexibility when creating web crawlers. Ruia is powered by Python’s asyncio library along with aiohttp, enabling developers to perform concurrent network requests efficiently and scrape data from websites with minimal overhead. Ruia follows a “write less, run faster” philosophy, emphasizing concise code and streamlined spider development. -
9
--- IMPORTANT : This project has been moved to GitHub at https://github.com/clstoulouse/motu-client-python. Download the last version from the release page https://github.com/clstoulouse/motu-client-python/releases. --- Motu is a high efficient and robust Web Server which fills the gap between heterogeneous Data Providers to End Users. Motu handles, extracts and transforms oceanographic huge volumes of data without performance collapse. This client enables to extract and download data through a python command line Indesol project sample: http://www.indeso.web.id/indeso_wp/index.php/faq/30-6-how-to-write-and-run-the-script-to-download-indeso-met-ocean-data
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
GoogleScraper
Python tool for scraping search engine results from many providers
GoogleScraper is a Python-based tool designed to automatically collect and process search engine results from multiple providers. It enables developers and researchers to programmatically query search engines and extract useful information such as links, titles, and result descriptions. GoogleScraper supports several major search engines and can be used to gather structured datasets from search result pages for further analysis. It provides two different scraping approaches: sending direct... -
11
Photon
Incredibly fast crawler designed for OSINT
Photon is an extremely fast web crawler built specifically for OSINT and reconnaissance use cases. It is designed to extract URLs, endpoints, files, and other intelligence artifacts from target websites with minimal overhead. The crawler prioritizes speed and breadth, making it suitable for mapping web attack surfaces and discovering hidden resources. Photon is commonly used during early reconnaissance phases to build a comprehensive inventory of reachable assets. ... -
12
Requests-HTML
Pythonic HTML Parsing for Humans
This library intends to make parsing HTML (e.g. scraping the web) as simple and intuitive as possible. When using this library you automatically get full JavaScript support! (Using Chromium, thanks to puppeteer) CSS Selectors (a.k.a jQuery-style, thanks to PyQuery). XPath Selectors, for the faint of heart. Mocked user-agent (like a real web browser). Automatic following of redirects. Connection–pooling and cookie persistence. -
13
Xplico
Xplico is a Network Forensic Analysis Tool (NFAT)
Xplico is a Network Forensic Analysis Tool (NFAT). The goal of Xplico is extract from an internet traffic capture the applications data contained. For example, from a pcap file Xplico extracts each email (POP, IMAP, and SMTP protocols), all HTTP contents, each VoIP call (SIP, MGCP, MEGACO, RTP), IRC, WhatsApp... Xplico is able to classify more than 140 (application) protocols. Xplico cam be used as sniffer-decoder if used in "live mode" or in conjunction with netsniff-ng. Xplico is used... -
14
mzitu
Python crawler that downloads image galleries and analyzes titles
mzitu is a Python-based web crawling project designed to automatically download and organize image galleries from a specific photography site. It demonstrates how to build a scraper that navigates gallery pages, retrieves image links, and saves the images locally in a structured directory layout. It focuses on automating the collection of large sets of images by programmatically parsing page content and iterating through gallery entries. mzitu also includes a simple analysis script that... -
15
WeChatSogou
Python library to crawl and retrieve data from WeChat accounts
WechatSogou is an open source Python library designed to retrieve data from WeChat official accounts by using the Sogou WeChat search service as its data source. It provides developers with a programmatic way to search for public accounts and collect article information without manually browsing the search interface. It functions as a crawler interface that sends requests to the search engine, retrieves results, and converts the returned pages into structured data that can be used in... -
16
Zeus Scanner
Advanced reconnaissance utility
Zeus-Scanner is an advanced reconnaissance utility designed to simplify web application reconnaissance for security testing. It combines URL parsing, search engine querying, crawling, proxy support, and vulnerability assessment workflows in one tool. The scanner can work with multiple search engines, extract URLs from Google ban and webcache URLs, and parse robots.txt or sitemap.xml files. It also supports proxy configurations, Tor proxy compatibility, and Tor browser emulation for flexible routing during authorized assessments. ... -
17
GreenOdoo
Portable Odoo (formerly OpenERP) for windows and linux x64
Portable Odoo (formerly OpenERP) Portable Odoo (formerly OpenERP) for windows and linux x64. Usage Extract the zip file and run start.bat( for windows) or start.sh (for linux) file. Open Brower and visit http://127.0.0.1:8069 Soure Repository https://github.com/buke/GreenOdoo Author: wangbuke <wangbuke@gmail.com> -
18
Cloud Export is a tool to automatically extract your data from web applications and save it to your local file system for backup purposes, but more extensive than Google Takeout. Plans are based on http://www.dataliberation.org.
-
19
Ymap - Yeast Mapping Analysis Pipeline
Pipeline for large-scale genome changes analysis of genome datasets.
The active use repository has migrated over to: https://github.com/darrenabbey/ymap The repository here was errantly created with some large binary files included. Attempts to extract the files from the history here have failed. A copy of the history was successfully scrubbed and then hosted at github. -------- Eukaryotic pathogens have complicated and dynamic genomes. To facilitate analysis of copy number variations (CNV), single nucleotide polymorphisms (SNPs), and loss of... -
20
Penetration-Testing-Toolkit v1.0
A web interface for various penetration testing tools
Penetration-Testing-Toolkit is a web based project to automate Scanning a network,Exploring CMS, Generating Undectable metasploit payload, DNS-Queries, IP related informations, Information Gathering, Domain related info etc -
21
Spondulas
Spondulas is browser emulator designed to retrieve web pages for hunti
Spondulas is browser emulator and parser designed to retrieve web pages for hunting malware. It supports generation of browser user agents, GET/POST requests, and SOCKS5 proxy. It can be used to parse HTML files sent via e-mail. Monitor mode allows a website to be monitored at intervals to discover changes in DNS or content over time. Autolog mode creates an investigation file that documents redirection chains. The retrieved web pages are parsed for links and reported to an output file. More... -
22
Framework for making Windows applications that are one .exe file in AutoHotKey_L,C++,C#, VB.NET,Java,Groovy,Common Lisp,Nemerle,Ruby,Python,PHP,Lua,Tcl,Perl,Jint,S#,WSH VBScript,HTML/JavaScript/CSS,COM, PowerShell without compiling . For .NET 4.
-
23
To create a framework to extract Web data and store in local RDBMS, to generate assessment reports on quality of the data being extract, and to publish the quality reports on the Web.
-
24
PySMBSearch is a crawler and search engine for SMB shares. It consists of a crawler script, which creates an index and stores it in an SQL database, and a CGI script that can be used to extract queries from the database.
-
25
litersta
Litersta - textual analytics - software
...Unlike database fields, which are easily queried, text contains unstructured data that must be parsed for key objects that can be transformed in to powerful metrics. Litersta - textual analytics - software leverages statistical algorithms to programmatically locate, and extract, overall document sentiment, word frequencies, and document similarities. The Litersta web application runs locally on your server - behind your firewall. This strategy keeps your data confidential and secure.
