Search Results for "data capture framework"
Sort By:
Showing 143 open source projects for "data capture framework"
View related business solutions-
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
1
Django REST framework
Powerful and flexible toolkit for building Web APIs
Django REST framework is a powerful and flexible toolkit for building Web APIs. Some reasons you might want to use REST framework: The Web browsable API is a huge usability win for your developers. Authentication policies including packages for OAuth1a and OAuth2. Serialization that supports both ORM and non-ORM data sources. Customizable all the way down - just use regular function-based views if you don't need the more powerful features. -
2
Synthetic Data Vault (SDV)
Synthetic Data Generation for tabular, relational and time series data
The Synthetic Data Vault (SDV) is a Synthetic Data Generation ecosystem of libraries that allows users to easily learn single-table, multi-table and timeseries datasets to later on generate new Synthetic Data that has the same format and statistical properties as the original dataset. Synthetic data can then be used to supplement, augment and in some cases replace real data when training Machine Learning models. Additionally, it enables the testing of Machine Learning or other data dependent... -
3
Union Pandera
Light-weight, flexible, expressive statistical data testing library
The open-source framework for precision data testing for data scientists and ML engineers. Pandera provides a simple, flexible, and extensible data-testing framework for validating not only your data but also the functions that produce them. A simple, zero-configuration data testing framework for data scientists and ML engineers seeking correctness. Access a comprehensive suite of built-in tests, or easily create your own validation rules for your specific use cases. ... -
4
OpenWPM
A web privacy measurement framework
OpenWPM is a web privacy measurement framework that makes it easy to collect data for privacy studies on a scale of thousands to millions of websites. OpenWPM is built on top of Firefox, with automation provided by Selenium. It includes several hooks for data collection. Check out the instrumentation section below for more details. OpenWPM is tested on Ubuntu 18.04 via TravisCI and is commonly used via the docker container that this repo builds, which is also based on Ubuntu. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. -
6
Kedro
A Python framework for creating reproducible, maintainable code
Kedro is an open sourced Python framework for creating maintainable and modular data science code. Provides the scaffolding to build more complex data and machine-learning pipelines. In addition, there's a focus on spending less time on the tedious "plumbing" required to maintain data science code; this means that you have more time to solve new problems. Standardises team workflows; the modular structure of Kedro facilitates a higher level of collaboration when teams solve problems together. ... -
7
Meridian
Meridian is an MMM framework
...The framework provides a robust foundation for constructing in-house MMM pipelines capable of handling both national and geo-level data, with built-in support for calibration using experimental data or prior knowledge. Meridian uses the No-U-Turn Sampler (NUTS) for Markov Chain Monte Carlo (MCMC) sampling to produce statistically rigorous results, and it includes GPU acceleration to significantly reduce computation time. -
8
Douyin TikTok Download API
Douyin TikTok Download API
Use the official interface to capture Douyin|TikTok data, support API calls, Web portals, and batch analysis. Fast, asynchronous, free, open source, ad-free, long-term maintenance. This project is based on PyWebIO , FastAPI , HTTPX , a fast and asynchronous Douyin / TikTok data crawling tool, and realizes online batch parsing and downloading of watermark-free videos or atlases through the web, data crawling API, and iOS shortcut instructions for watermark-free download and other functions. ... -
9
WTForms
A flexible forms validation and rendering library for Python
...Flask-WTF integrates with the Flask framework. It can automatically load data from the request, uses Flask-Babel to translate based on user-selected locale, provides full-application CSRF, and more. -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
10
Dash
Build beautiful web-based analytic apps, no JavaScript required
Dash is a Python framework for building beautiful analytical web applications without any JavaScript. Built on top of Plotly.js, React and Flask, Dash easily achieves what an entire team of designers and engineers normally would. It ties modern UI controls and displays such as dropdown menus, sliders and graphs directly to your analytical Python code, and creates exceptional, interactive analytics apps. Dash apps are very lightweight, requiring only a limited number of lines of Python or... -
11
NVIDIA Warp
A Python framework for accelerated simulation, data generation
NVIDIA Warp is a high-performance Python framework developed by NVIDIA for building and accelerating simulation, graphics, and physics-based workloads using GPU computing. It enables developers to write kernel-level code in Python that is automatically compiled into efficient CUDA kernels, combining ease of use with near-native performance. The framework is designed for applications such as robotics, reinforcement learning, physical simulation, and differentiable computing, where performance... -
12
Behaviour Suite Reinforcement Learning
bsuite is a collection of carefully-designed experiments
...Each experiment in bsuite is meticulously designed to capture key challenges in RL, such as exploration, credit assignment, and stability. The framework supports automated logging and analysis, generating standardized output compatible with Jupyter notebooks for streamlined evaluation. It also integrates easily with existing RL libraries and can be used locally or via cloud computing platforms, including Google Cloud. -
13
learn_python3_spider
Python crawler tutorial series, learning Python crawlers from 0 to 1
learn_python3_spider is a tutorial and example-code collection for learning web scraping with Python 3 from beginner to advanced topics. It introduces request capture, browser and mobile traffic inspection, and core scraping libraries such as Requests, BeautifulSoup, Selenium, Appium, and Scrapy. Lessons also cover JSON parsing, regular expressions, concurrency, proxy usage, login handling, and CAPTCHA-related techniques. Data can be stored with CSV, MySQL, or MongoDB examples. More advanced material explores anti-scraping defenses, JavaScript analysis, WebSocket data, and distributed crawlers. ... -
14
Frappe
Low code web framework for real world applications
Frappe is a full-stack, low-code web framework written in Python and JavaScript, used to build scalable and modular enterprise applications. It powers ERPNext and includes tools for REST APIs, user management, document modeling, workflows, and real-time updates. Frappe uses a "model-view-controller" approach with its own ORM and frontend system, enabling rapid development without sacrificing control or performance. -
15
Marin
Framework for research and development of foundation models
Marin is an open-source research platform and community for developing foundation models through transparent, reproducible experimentation. It covers the complete model-building pipeline from data curation and filtering through tokenization, pretraining, post-training, and evaluation. Experiments and decisions are documented as they occur, including unsuccessful approaches. The framework is primarily used for large language models but has also supported audio-text, DNA, and protein modeling research. Experiments are expressed as dependent steps that execute in topological order, enabling reproducible training workflows. ... -
16
pywebview
Build GUI for your Python program with JavaScript, HTML, and CSS
pywebview is a lightweight cross-platform wrapper around a webview component that allows to display HTML content in its own native GUI window. It gives you power of web technologies in your desktop application, hiding the fact that GUI is browser based. You can use pywebview either with a lightweight web framework like Flask or Bottle or on its own with a two way bridge between Python and DOM. pywebview uses native GUI for creating a web component window: WinForms on Windows, Cocoa on macOS... -
17
Graphene
GraphQL in Python Made Easy
Graphene is a Python library for building GraphQL APIs fast and easily, using a code-first approach. Instead of writing GraphQL Schema Definition Langauge (SDL), Python code is written to describe the data provided by your server. Graphene helps you use GraphQL effortlessly in Python, but what is GraphQL? GraphQL is a data query language developed internally by Facebook as an alternative to REST and ad-hoc webservice architectures. With Graphene you have all the tools you need to... -
18
easyquant
Stock quantification framework to support market acquisition
easyquant is a Python quantitative-trading framework built on top of easytrader for execution and easyquotation for market data. Its event-driven architecture is inspired by vn.py and routes market updates into user-defined strategies. The default configuration can stream broad Sina market data at roughly one-second intervals. Alternative quotation sources can be substituted, including selected depth and fund data supported by easyquotation. -
19
SDGym
Benchmarking synthetic data generation methods
The Synthetic Data Gym (SDGym) is a benchmarking framework for modeling and generating synthetic data. Measure performance and memory usage across different synthetic data modeling techniques – classical statistics, deep learning and more! The SDGym library integrates with the Synthetic Data Vault ecosystem. You can use any of its synthesizers, datasets or metrics for benchmarking. -
20
Superduper
Superduper: Integrate AI models and machine learning workflows
Superduper is a Python-based framework for building end-2-end AI-data workflows and applications on your own data, integrating with major databases. It supports the latest technologies and techniques, including LLMs, vector-search, RAG, and multimodality as well as classical AI and ML paradigms. Developers may leverage Superduper by building compositional and declarative objects that out-source the details of deployment, orchestration versioning, and more to the Superduper engine. ... -
21
CoreNet
CoreNet: A library for training deep neural networks
CoreNet is Apple’s internal deep learning framework for distributed neural network training, designed for high scalability, low-latency communication, and strong hardware efficiency. It focuses on enabling large-scale model training across clusters of GPUs and accelerators by optimizing data flow and parallelism strategies. CoreNet provides abstractions for data, tensor, and pipeline parallelism, allowing models to scale without code duplication or heavy manual configuration. ... -
22
EasyR1
An Efficient, Scalable, Multi-Modality RL Training Framework
EasyR1 is a streamlined training framework for building “R1-style” reasoning models from open-source LLMs with minimal boilerplate. It focuses on the full reasoning stack—data preparation, supervised fine-tuning, preference or outcome-based optimization, and lightweight evaluation—so you can iterate quickly on chain-of-thought–heavy tasks. The project’s philosophy is practicality: sensible defaults, one-command recipes, and compatibility with popular base models let you stand up experiments without wrestling infrastructure. ... -
23
Smallpond
A lightweight data processing framework built on DuckDB and 3FS
smallpond is a lightweight distributed data processing framework built by DeepSeek, designed to scale DuckDB workloads over clusters using their 3FS (Fire-Flyer File System) backend. The idea is to preserve DuckDB’s fast analytics engine but lift it from single-node to multi-node settings, giving you the ability to operate on large datasets (e.g. petabyte scale) without moving to a heavyweight system like Spark. -
24
ProtoMotions
ProtoMotions is a GPU-accelerated simulation and learning framework
ProtoMotions 3 is a GPU-accelerated framework for training physically simulated digital humans and humanoid robots. It provides modular environments and learning components for animation, robotics, and reinforcement learning research. Large motion datasets such as AMASS and BONES can be divided across multiple GPUs for scalable policy training. A built-in PyRoki workflow retargets human motion data to different robot bodies with a single command. -
25
PaddleX
PaddlePaddle End-to-End Development Toolkit
...When the model is trained, we need to divide the training set, the validation set and the test set. Therefore, we need to divide the above data. Using the paddlex command, the data set can be randomly divided into 70% training set, 20% validation set and 10% test set. If you use the PaddleX visualization client for model training, the data set division function is integrated in the client, and you do not need to use command division by yourself.
