Search Results for "data processing" - Page 2
Sort By:
Showing 125 open source projects for "data processing"
View related business solutions-
$300 in Free Credit Towards Top Cloud ServicesStart your project in minutes. After credits run out, 20+ products include free monthly usage. Only pay when you're ready to scale.
-
Gemini 3 and 200+ AI Models on One PlatformBuild generative AI apps with Vertex AI. Switch between models without switching platforms.
-
1
Web data extraction (web data mining, web scraping) tool. It leverages well proved XML and text processing techologies in order to easely extract useful data from arbitrary web pages.
-
2
mendelson OFTP2
Implementation of the OFTP2 protocol (RFC 5024)
The mendelson OFTP2 is a JAVA-based open source OFTP2 solution. It contains a logging- and configuration UI and supports encryption, digital signatures, message compression, TLS, certificate exchange, message routing, mail notification -
3
apache-logs-to-mysql
Apache Log Parser and Data Normalization Application
Apache Log Parser and Data Normalization Application Python handles File Processing & MySQL handles Data Processing ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis. Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0. 4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution. ... -
4
uriparser
RFC 3986 URI parsing and processing libary
PLEASE NOTE that we are in the process of moving to GitHub: https://github.com/uriparser/uriparser uriparser is a strictly RFC 3986 compliant URI parsing library written in C89. uriparser is cross-platform, fast, supports Unicode and is licensed under the New BSD license. PLEASE NOTE that we are in the process of moving to GitHub: https://github.com/uriparser/uriparser -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
queXF
Web based, Open Source alternative to Remark OMR or Teleform
queXF, a CADE (Computer Assisted Data Entry) Tool, processes filled paper forms that were created in queXML, such as survey questionnaires. queXF can be used as a web based, Open Source alternative to programs such as Cardiff Teleform and Remark OMR. -
6
uweb browser: unlimited power
minimal suckless android web browser with unlimited power
...and .js files as commands). - user-defined site-specific JS/CSS/HTML/preprocessing. - Online play/preview/preprocess for downloadable resources. - Multiple type profiles: switch any data including logins/config orthogonally - web automation, crontab (alarm clock) -
7
MBR Bulk WP Detector
A free WP plugin that lets you check unlimited URLs
MBR Bulk WP Detector is a free WordPress plugin that lets you check unlimited URLs right from your own dashboard. No subscriptions, no URL limits, and your data stays completely private on your server. What Can You Do With It? The basics are simple: Paste a list of URLs (or upload a CSV file), click a button, and boom—you’ve got a clear breakdown of which sites are running WordPress and which aren’t. But it gets better… Turn on Deep Scan mode, and you’ll also discover what... -
8
crawly
High-level web crawling and scraping framework for Elixir apps
Crawly is a high-level application framework for crawling websites and extracting structured data using the Elixir programming language. It provides a complete environment for building web crawlers that systematically visit pages, collect information, and transform that data into structured formats for further processing. Crawly is designed for tasks such as data mining, information processing, and building historical archives of web content. ... -
9
mlscraper
ML-based HTML scraper that learns extraction rules from examples
mlscraper is a Python library designed to automatically extract structured data from HTML pages without requiring developers to manually write CSS selectors or XPath rules. Instead of defining extraction logic by hand, users provide a few examples of the data they want to retrieve from a webpage. It analyzes those examples within the HTML document and determines patterns or rules that can be used to extract the same type of information from similar pages. Once trained, the generated scraper... -
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
10
pspider
Simple Python framework for building multithreaded web crawlers
PSpider is a lightweight web crawling framework written in Python designed to simplify the development of custom web spiders. It focuses on providing an easy-to-understand architecture while still supporting concurrent crawling for improved performance. It uses a multithreaded model that separates the crawling workflow into several components responsible for fetching, parsing, and saving data. Tasks are managed through queues, allowing different parts of the crawler to process work... -
11
SiteofSiteIDE
Static site IDE is a Static Site Generators aka Static Site Editor
...In reality, a minimum of php/asp code is used to establish the browser language and cookie management. Support for the GDPR is included as an example (it should be modified according to the processing of the website owner's data). -
12
Offline First Database Comparison
An implementation of the exact same app in Firestore, AWS Datastore
...Also, you might want to create new metrics that better represent how you would use the respective database. WatermelonDB and the RxDB-LokiJS project use the LokiJS database as storage, which is an in memory database that regularly persists the data to IndexedDB either on interval, or when the browser tab is closed. By doing so, less slow IndexedDB transaction are used. Keeping and processing the data in memory has the benefit of being much faster. -
13
Abot
Fast and flexible C# framework for building customizable web crawlers
...It focuses on speed, flexibility, and extensibility while handling the complex low-level tasks involved in web crawling. It manages essential components such as multithreading, HTTP requests, scheduling, and link parsing so developers can focus on processing the collected data. Abot follows a modular architecture that allows developers to customize nearly every stage of the crawl process by implementing or replacing core interfaces. Abot exposes an event-driven model that enables applications to react to crawling events such as page completion or crawl restrictions. It also provides configuration options that control crawling behavior including concurrency limits, crawl delays, and request parameters. ... -
14
TRACARDI - Customer Data Platform
TRACARDI free open-source customer data platform
TRACARDI is easy to use and free GUI for Apache Unomi. Unomi is an open source Customer Data Platform that allows anyone to collect user profiles and manage them in a very robust way. TRACARDI with is API first approach enables you to collect data from multiple channels. Regardless if it is web site, mobile app or CRM system open Api let you send data for further processing. Integrate data into one consistent user profile. -
15
ruia
Async Python framework for fast and flexible web scraping spiders
...It also supports middleware and plugin systems that allow customization of request handling, response processing, and additional functionality. -
16
csv-parser
Streaming csv parser inspired by binary-csv that aims to be faster
csv-parser is a streaming CSV parsing library for Node.js designed for efficiency and correctness. It implements the stream API native to Node, allowing you to pipe a file or readable stream into the parser and process each row (as a JavaScript object or array) as soon as it's parsed — which is crucial for handling large CSV files without loading them entirely into memory. The parser handles standard CSV semantics including quoted fields, variable delimiters, escape sequences, and optional... -
17
CSSBox
Pure Java HTML / CSS rendering engine
CSSBox is an (X)HTML/CSS rendering engine written in pure Java. Its primary purpose is to provide a complete information about the rendered page suitable for further processing. However, it also allows displaying the rendered document. -
18
mod_psldap
Apache LDAP Directory Manager
mod_psldap is an Apache module for leveraging LDAP services built on the OpenLDAP library and the Apache APIs, to include web based A&A, web based updates to the LDAP store, server-side XSLT processing, and session management across servers. -
19
Fire RSS Reader
RSS Reader Firefox addon extension
Your Reader of RSS feeds for browser Firefox Get it - https://addons.mozilla.org/en-US/firefox/addon/fire-rss-reader/ How it works - https://www.youtube.com/watch?v=0bx-gqBJ7uo To run and build browser extension from cmd or terminal type: - npm install --global web-ext - cd my/path/folder/FireRSS - web-ext run - web-ext build -
20
iText®, a JAVA PDF library
PDF Library for Developers
iText is an open-source PDF library available for Java and .NET (C#). iText allows you to effortlessly generate and manipulate standards-compliant PDF documents with a powerful and feature-rich SDK. With iText, you can create archivable and accessible PDFs, split and merge documents, fill and flatten forms, digitally sign documents, and more. iText add-ons enable additional functionality, such as PDF creation from HTML templates, secure redaction, OCR, and much more. The latest... -
21
mzitu
Python crawler that downloads image galleries and analyzes titles
...Using text segmentation and frequency analysis, the project can create a word cloud representing common keywords found in the dataset. This makes the repository both a scraping example and a small data analysis experiment built around the collected content. Overall, mzitu serves as a learning-oriented implementation of Python web scraping, data processing, and visualization techniques. -
22
Command-line/Ant-task/embeddable text file preprocessor. Macros, flow control, expressions. Recursive directory processing. Extensible in Java to display data from any data sources (as database). Can generate complete homepages (tree of HTML-s, images, etc.)
-
23
Twitter Intelligence
Twitter Intelligence OSINT project performs tracking and analysis
...This project is a Python 3.x application. The package dependencies are in the file requirements.txt. Run that command to install the dependencies. SQLite is used as the database. Tweet data is stored on the Tweet, User, Location, Hashtag, HashtagTweet tables. The database is created automatically. analysis.py performs analysis processing. User, hashtag, and location analyzes are performed. You must write Google Map API Key in setting.py to display Google Maps. -
24
gain
Asyncio-based Python framework for building fast web crawling spiders
Gain is a Python web crawling framework designed to simplify the process of building efficient and scalable web scrapers. It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and... -
25
survol
RDF-based framework monitoring business systems activity
A Python agent and a web interface aiming to help the analysis and investigation of a legacy application. A set of machines, processes, databases, programs etc ... all communicating with each other, manipulating your data, and whose software architecture has become, with time, complicated, difficult to understand, and undocumented. Data are aggregated with an RDF inference engine, creating a global vision of the business information processing.
