Search Results for "data processing" - Page 2
Sort By:
Showing 61 open source projects for "data processing"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
1
Fondant
Production-ready data processing made easy and shareable
Fondant is a modular, pipeline-based framework designed to simplify the preparation of large-scale datasets for training machine learning models, especially foundation models. It offers an end-to-end system for ingesting raw data, applying transformations, filtering, and formatting outputs—all while remaining scalable and traceable. Fondant is designed with reproducibility in mind and supports containerized steps using Docker, making it easy to share and reuse data processing components. It’s built for use in research and production, empowering data scientists to streamline dataset curation and preprocessing workflows efficiently. -
2
airda
airda(Air Data Agent
airda(Air Data Agent) is a multi-smart body for data analysis, capable of understanding data development and data analysis needs, understanding data, generating data-oriented queries, data visualization, machine learning and other tasks of SQL and Python codes. -
3
Mara Pipelines
A lightweight opinionated ETL framework, halfway between plain scripts
This package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code. PostgreSQL as a data processing engine. Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines. GNU make semantics. -
4
text-dedup
All-in-one text de-duplication
text-dedup is a Python library that enables efficient deduplication of large text corpora by using MinHash and other probabilistic techniques to detect near-duplicate content. This is especially useful for NLP tasks where duplicated training data can skew model performance. text-dedup scales to billions of documents and offers tools for chunking, hashing, and comparing text efficiently with low memory usage. It supports Jaccard similarity thresholding, parallel execution, and flexible... -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
GXSM
Scanning Probe Microscopy Controller and Data Visualization Software
GXSM -- Gnome X Scanning Microscopy: A multi-channel image and vector-probe data acquisition and visualization system designed for SPM techniques (STM,AFM..), but also SPA-LEED/LEED/LEEM data analysis. A plug-in interface allows any user add-on data-processing and special hardware and instrument support. Latest: NC-AFM and related explorative methods as SQDM can be configured. High-Speed external PAC-PLL hardware option with digital DSP link. -
6
Wooey
A Django app that creates automatic web UIs for Python scripts
Wooey is a simple web interface to run command line Python scripts. Think of it as an easy way to get your scripts up on the web for routine data analysis, file processing, or anything else. The project was inspired by how simply and powerfully sandman could expose users to a database and by how Gooey turns ArgumentParser-based command-line scripts into WxWidgets GUIs. Originally two separate projects (Django-based djangui by Chris Mitchell and Flask-based Wooey by Martin Fitzpatrick) it has been merged to combine our efforts. ... -
7
Padasip
Python Adaptive Signal Processing
Padasip (Python Adaptive Signal Processing) is a Python library tailored for adaptive filtering and online learning applications, particularly in signal processing and time series forecasting. It includes a variety of adaptive filter algorithms such as LMS, RLS, and their variants, offering real-time adaptation to changing environments. The library is lightweight, well-documented, and ideal for research, prototyping, or teaching purposes. Padasip supports both supervised and unsupervised... -
8
Forecasting Best Practices
Time Series Forecasting Best Practices & Examples
...Rather than creating implementations from scratch, we draw from existing state-of-the-art libraries and build additional utilities around processing and featuring the data, optimizing and evaluating models, and scaling up to the cloud. The examples and best practices are provided as Python Jupyter notebooks and R markdown files and a library of utility functions. -
9
Spectral Python
A python module for hyperspectral image processing
Spectral Python (SPy) is a python package for reading, viewing, manipulating, and classifying hyperspectral image (HSI) data. SPy includes functions for clustering, dimensionality reduction, supervised classification, and more. -
PRTG Catches Network Issues Before They Cause DowntimeReactive troubleshooting usually means hearing about a problem from frustrated users, not your monitoring tool. PRTG sets threshold-based alerts across devices, servers and applications, notifying your team by email, SMS or push the moment a metric crosses a set limit. That means catching a failing disk or overloaded server before it becomes an outage and getting time back from firefighting. Start a free trial and set your first alerts today.
-
10
SCiDA Pro
Program for solar cell production data analysis
The purpose of the SCiDA Pro program is to help with processing solar cell production data. It has the following features: - Able to handle large data sets in a fast way (e.g. plotting 100k cell data takes a few seconds) - Easy data filtering - Easy generation of a data summary report - Extensive data plotting features - Cross-platform (Windows/Linux/MacOS) - Supports multiple languages -
11
nonechucks
Deal with bad samples in your dataset dynamically
...Or what if your dataset is a folder full of scanned PDFs that you have to OCRize, and then run a language detector on the resulting text, because you want only the ones that are in English? Or maybe you have an AlternateIndexSampler, and you want to be able to move to dataset[6] after dataset[4] fails while attempting to load! PyTorch's data processing module expects you to rid your dataset of any unwanted or invalid samples before you feed them into its pipeline, and provides no easy way to define a "fallback policy" in case such samples are encountered during dataset iteration. -
12
LaueTools
open source python packages for X-ray MicroLaue Diffraction analysis
LaueTools is an open-source project for white beam Laue x-ray microdiffraction data analysis including tools in image processing, peaks searching & indexing, crystal structure solving (orientation & strain) and data & grain mapping visualisation. Python 3 Code and new features are now at: https://gitlab.esrf.fr/micha/lauetools -
13
Wally
Distributed Stream Processing
...Provide high-performance & low-latency data processing. Be portable and deploy easily (i.e., run on-prem or any cloud). Manage in-memory state for the application. Allow applications to scale as needed, even when they are live and up-and-running. The primary API for Wally is written in Pony. Wally applications are written using this Pony API. -
14
AI learning
AiLearning, data analysis plus machine learning practice
We actively respond to the Research Open Source Initiative (DOCX) . Open source today is not just open source, but datasets, models, tutorials, and experimental records. We are also exploring other categories of open source solutions and protocols. I hope you will understand this initiative, combine this initiative with your own interests, and do what you can. Everyone's tiny contributions, together, are the entire open source ecosystem. We are iBooker, a large open-source community,... -
15
Data analysis for ADEXL
Create Excel reports from results of Cadence ADE-XL simulations
*** Required files from "Excel COM API for Python" project *** The scripts allows post processing of data saved in CSV format from "Detail - Transposed" view of ADE-XL -
16
BioImageXD - free open source software for analysis, processing and 3D rendering of multi dimensional, multi data channel, time series image data from microscopy and other sources.
-
17
scipion-xmipp
Image processing framework to integrate EM software packages.
Scipion is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy (3DEM). It integrates several software packages and presents an unified interface for both biologists and developers. Scipion allows to execute workflows combining different software tools, while taking care of formats and conversions. Additionally, all steps are tracked and can be reproduced later on. Xmipp is a well-known package in the EM image processing. It is integrated... -
18
iMir
Integrated pipeline for HT miRNA-Seq data analysis
Processing of smallRNA-Seq data to gather biologically relevant information requires application of multiple statistical and bioinformatics tools from different sources, each focusing on a specific step of the analysis pipeline. The analytical workflow can be challenging for the continuous interventions by the operator, a critical factor when large numbers of datasets need to be analyzed at once. -
19
Larch: Data Analysis for X-ray Spectra
Data Processing and Analysis for X-ray Spectroscopy and More
Larch is a scientific data processing language that is designed to be easy to use for novices and complete enough for advanced data processing and analysis. Larch provides a wide range of functionality for dealing with arrays of scientific data, and basic tools to make it easy to use and organize complex data. Larch has been primarily developed for dealing with x-ray spectroscopic and scattering data, especially the kind of data collected at modern synchrotrons and x-ray sources. ... -
20
Pathomx
Workflow-based data analysis built on IPython
Pathomx is a workflow-based tool for the analysis and visualisation of experimental data. Initially created as a tool for metabolomic data analysis is has been extended and can now be used for any scientific and non-scientific data analysis. -
21
OpenCMISS-Zinc
A Library for Interactive Modelling and Visualisation
...Models are represented in Zinc as mathematical fields defined over domains, including finite elements with support for high-order basis functions, complex parameter mappings and time variation, and image-based fields. Further fields can be defined by mathematical expressions and algorithms on existing fields, including image processing filters. Zinc model data structures are dynamic, supporting interactive applications which programmatically create, destroy and modify parts of models. -
22
allink
Software for data analysis, image processing, simulations, solver.
Collection of utilities based on two basics classes: Matematica and VarData. Matematica) performs math operations on vectors and matrices for smoothing, interpolation, convolution, image processing... VarData) manipulate a structure of points connected by links. Addraw) openGL engine. ElPoly) analyze mechanical properties of polymer and membrane like structures. Addyn) perform molecular dynamics and Monte Carlo simulations and has a solver for 4th oder PDE. Avvis) perform all the operation of Matematica on different sets of data visualized on a Qt graphical interface. ... -
23
Pyphant is a framework for scientific data analysis. It features a flexible plugin architecture allowing for the application in many different fields. For the analysis a graph of processing steps is constructed, that can be applied via the cli as well.
-
24
totaldepth
Petrophysical data processing and presentation.
TotalDepth is an Open Source, cross platform, software collection that can process petrophysical data from the oil field such as wireline logs, seismic data and so on. Conventional, proprietary, software for petrophysical data tends to be expensive to licence, restrictive, slow to develop for and tied to expensive hardware. TotalDepth changes all of that. TotalDepth is open and cross-platform, and produces results straight to the bowser. TotalDepth supports such technologies such as... -
25
