Search Results for "s3"
Sort By:
Showing 41 open source projects for "s3"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
s3-client
Sample python script to work with Amazon S3
Example Python script to work with S3. -
2
AWS Data Wrangler
Pandas on AWS, easy integration with Athena, Glue, Redshift, etc.
...Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON, and EXCEL). Built on top of other open-source projects like Pandas, Apache Arrow and Boto3, it offers abstracted functions to execute usual ETL tasks like load/unload data from Data Lakes, Data Warehouses, and Databases. Convert the column name to be compatible with Amazon Athena and the AWS Glue Catalog. Run a query against AWS CloudWatchLogs Insights and convert the results to Pandas DataFrame. ... -
3
smart_open
Utils for streaming large files (S3, HDFS, gzip, bz2...)
smart_open is a Python library for streaming very large files from local or remote storage through a familiar file-like interface. It acts as an extended replacement for Python's built-in open function. Supported backends include Amazon S3, Google Cloud Storage, Azure Blob Storage, HDFS, WebHDFS, HTTP, HTTPS, SFTP, and local filesystems. Data can be read or written without first downloading an entire remote object to disk. Transparent compression and decompression are supported for multiple common formats. Standard operations such as context managers, iteration, and supported seek behavior remain available. ... -
4
SageMaker TensorFlow Training Toolkit
Toolkit for running TensorFlow training scripts on SageMaker
...After creating a SageMaker Model, you can use it to create SageMaker Batch Transform Jobs for offline inference, or create SageMaker Endpoints for real-time inference. A SageMaker Model contains references to a model.tar.gz file in S3 containing serialized model data, and a Docker image used to serve predictions with that model. A Batch Transform job runs an offline-inference job using your TensorFlow Serving model. Input data in S3 is converted to HTTP requests, and responses are saved to an output bucket in S3. -
Build Data Resilience - Take the Assessment TodayIs your recovery strategy as strong as you think? Take this quick self-assessment to check your recovery readiness and gain tailored insights. In only 2 minutes, you'll learn where you fall on the recovery readiness scale.
-
5
django-dbbackup
Management commands to help backup and restore your project database
django-dbbackup is a Django management command extension for backing up and restoring databases and media files. It supports multiple storage backends including local file systems, Amazon S3, and Dropbox. Ideal for automated backup strategies, it simplifies the integration of backup logic into Django projects. -
6
Amazon Braket Python Schemas
A library that contains schemas for Amazon Braket
Amazon Braket Python Schemas is an open source library that contains the schemas for Braket, including intermediate representations (IR) for Amazon Braket quantum tasks and offers serialization and deserialization of those IR payloads. Think of the IR as the contract between the Amazon Braket SDK and Amazon Braket API for quantum programs. Schemas for the S3 results of each quantum task. Schemas for the device capabilities of each device. The preferred way to get Amazon Braket Python Schemas is by installing the Amazon Braket Python SDK, which will pull in the schemas. You can install from source by cloning this repository and running a pip install command in the root directory of the repository. ... -
7
Deep Lake
Data Lake for Deep Learning. Build, manage, and query datasets
...It can be deployed locally or in the cloud, and it enables you to store all of your data in one place, ranging from simple annotations to large videos. Deep Lake is used by Google, Waymo, Red Cross, Omdena, Yale, & Oxford. Use one API to upload, download, and stream datasets to/from AWS S3/S3-compatible storage, GCP, Activeloop cloud, or local storage. Store images, audios and videos in their native compression. Deeplake automatically decompresses them to raw data only when needed, e.g., when training a model. Treat your cloud datasets as if they are a collection of NumPy arrays in your system's memory. Slice them, index them, or iterate through them. -
8
DVC
Data Version Control | Git for Data & Models
...It is designed to handle large files, data sets, machine learning models, and metrics as well as code. Version control machine learning models, data sets and intermediate files. DVC connects them with code and uses Amazon S3, Microsoft Azure Blob Storage, Google Drive, Google Cloud Storage, Aliyun OSS, SSH/SFTP, HDFS, HTTP, network-attached storage, or disc to store file contents. Version control machine learning models, data sets, and intermediate files. DVC connects them with code and uses Amazon S3, Microsoft Azure Blob Storage, Google Drive, Google Cloud Storage, Aliyun OSS, SSH/SFTP, HDFS, HTTP, network-attached storage, or disc to store file contents. ... -
9
Cookiecutter Django
Framework for jumpstarting production-ready Django projects quickly
...Provides an optional basic ASGI setup for Websockets and an optional custom static build using Gulp and livereload. Send emails via Anymail (using Mailgun by default or Amazon SES if AWS is selected cloud provider, but switchable). Media storage using Amazon S3 or Google Cloud Storage. Docker support using docker-compose for development and production (using Traefik with LetsEncrypt support). Procfile for deploying to Heroku. Provides instructions for deploying to PythonAnywhere. You can run tests with unittest or pytest. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
CTFd
CTFs as you need them
...Challenge plugin architecture to create your own custom challenges. Static & Regex-based flags. Custom flag plugins. Unlockable hints. File uploads to the server or an Amazon S3-compatible backend. Limit challenge attempts & hide challenges. Automatic bruteforce protection. Individual and Team-based competitions. Have users play on their own or form teams to play together. Scoreboard with automatic tie resolution. Hide Scores from the public. Freeze Scores at a specific time. Scoregraphs comparing the top 10 teams and team progress graphs. ... -
11
AWS SDK for pandas
Easy integration with Athena, Glue, Redshift, Timestream, Neptune
aws-sdk-pandas (formerly AWS Data Wrangler) bridges pandas with the AWS analytics stack so DataFrames flow seamlessly to and from cloud services. With a few lines of code, you can read from and write to Amazon S3 in Parquet/CSV/JSON/ORC, register tables in the AWS Glue Data Catalog, and query with Amazon Athena directly into pandas. The library abstracts efficient patterns like partitioning, compression, and vectorized I/O so you get performant data lake operations without hand-rolling boilerplate. It also supports Redshift, OpenSearch, and other services, enabling ETL tasks that blend SQL engines and Python transformations. ... -
12
Agent S
Agent S: an open agentic framework that uses computers like a human
...Built to operate graphical user interfaces like a human, it allows AI agents to perceive screens, reason about tasks, and execute actions across macOS, Windows, and Linux systems. The latest version, Agent S3, surpasses human-level performance on the OSWorld benchmark, demonstrating state-of-the-art results in complex multi-step computer tasks. Agent S combines powerful foundation models (such as GPT-5) with grounding models like UI-TARS to translate visual inputs into precise executable actions. It supports flexible deployment via CLI, SDK, or cloud, and integrates with multiple model providers including OpenAI, Anthropic, Gemini, Azure, and Hugging Face endpoints. ... -
13
AWS Chalice
Python Serverless Microframework for AWS
...You define routes, event handlers, and background tasks in plain Python, and Chalice turns them into AWS Lambda functions wired to API Gateway, Amazon EventBridge schedulers, S3/SNS/SQS triggers, and more. A single command builds your app, bundles dependencies, generates infrastructure templates, and deploys to your account with sensible defaults. The framework includes local development tools so you can iterate on HTTP routes and inspect logs before shipping to the cloud. It also manages per-stage configuration, environment variables, IAM permissions scaffolding, and packaging optimizations to keep cold starts small. ... -
14
aws-devops-zero-to-hero
AWS zero to hero repo for devops engineers to learn AWS in 30 Days
...The README is structured as a day-by-day syllabus, starting with “Day 1: Introduction to AWS” and moving through IAM, EC2, VPC networking, security, DNS (Route 53), storage (S3), and many other core services. Each day mixes explanation with at least one concrete project or lab, such as deploying applications on EC2, designing secure VPCs, setting up CI/CD pipelines, or configuring CloudWatch monitoring. Later in the curriculum, you move into topics like CloudFormation, CodeCommit/CodePipeline/CodeBuild/CodeDeploy, Terraform on AWS, CloudTrail and Config for compliance, Elastic Load Balancing, and cloud migration strategies. ... -
15
PGHoard
PostgreSQL® backup and restore service
pghoard is a PostgreSQL backup and restore tool that provides encrypted, compressed, and cloud-optimized backups. Developed by Aiven, it supports streaming WAL archiving and full base backups to various cloud storage backends. pghoard is designed for reliability and fast disaster recovery in cloud-native PostgreSQL deployments. -
16
DocsGPT
Private AI agents, assistants and search over your own documents
DocsGPT is an open-source (MIT) platform for private AI agents, assistants and enterprise search over your own documents, built by Arc53. It ingests PDFs, Office files, scanned pages (OCR), web pages and repositories, syncs Google Drive, SharePoint, Confluence, GitHub and S3, and answers with citations to the source passage. Agents call tools, APIs and MCP servers with per-action approval; a visual builder chains AI steps, conditions and code into workflows; agents can produce DOCX, XLSX, PPTX and PDF files. Models: OpenAI, Anthropic, Google and others, or local via Ollama, llama.cpp or vLLM. Vectors: pgvector, Qdrant, Milvus, Elasticsearch, MongoDB Atlas or FAISS. ... -
17
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
18
aws-cli
Universal Command Line Interface for Amazon Web Services
The AWS CLI is the universal command-line interface for managing AWS services, automating tasks, and scripting cloud workflows. It exposes nearly every public API from EC2 and S3 to IAM, Lambda, and beyond, providing parity with the service SDKs in a tool you can run anywhere. Profiles, regions, single-sign-on, and credential helpers make it straightforward to switch contexts securely across accounts and environments. Its output controls and JMESPath querying let you slice, filter, and transform JSON responses directly in the shell, which is essential for automation. ... -
19
cloud_enum
Multi-cloud OSINT tool for discovering public cloud resources
cloud_enum is an open source reconnaissance and OSINT tool designed to discover publicly accessible cloud resources across major cloud providers. It focuses on enumerating assets in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform using keyword-based discovery techniques. It works by taking user-provided keywords and generating variations through mutation wordlists, then testing these combinations against common cloud service naming patterns. cloud_enum performs both... -
20
SQL Explorer
Easily share data across your company via SQL queries
SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can add to an existing Django site, or use as a standalone business intelligence tool. Quickly write and share SQL queries in a simple, usable SQL editor, preview the results in the browser, share links, download CSV, JSON, or Excel files (and even expose queries as API endpoints, if desired), and keep the information flowing! Comes with support for multiple... -
21
SDGym
Benchmarking synthetic data generation methods
The Synthetic Data Gym (SDGym) is a benchmarking framework for modeling and generating synthetic data. Measure performance and memory usage across different synthetic data modeling techniques – classical statistics, deep learning and more! The SDGym library integrates with the Synthetic Data Vault ecosystem. You can use any of its synthesizers, datasets or metrics for benchmarking. You also customize the process to include your own work. Select any of the publicly available datasets from the... -
22
esp32-ai
Running a 28.9M parameter LLM on an $8 microcontroller
esp32-ai is an experimental language-model project that runs a 28.9-million-parameter model entirely on an ESP32-S3 microcontroller. The quantized model occupies about 14.9 MB and generates text without sending data to a server. Most parameters remain in flash through a Per-Layer Embeddings design, while active computation uses SRAM and PSRAM. This memory layout allows the device to retrieve only the embedding rows required for each token. The implementation reaches roughly 9.5 tokens per second and can display generated words on a connected screen. ... -
23
Kedro
A Python framework for creating reproducible, maintainable code
Kedro is an open sourced Python framework for creating maintainable and modular data science code. Provides the scaffolding to build more complex data and machine-learning pipelines. In addition, there's a focus on spending less time on the tedious "plumbing" required to maintain data science code; this means that you have more time to solve new problems. Standardises team workflows; the modular structure of Kedro facilitates a higher level of collaboration when teams solve problems... -
24
AWS IoT Device SDK for Python
SDK for connecting to AWS IoT from a device using Python
...By connecting their devices to AWS IoT, users can securely work with the message broker, rules, and the device shadow (sometimes referred to as a thing shadow) provided by AWS IoT and with other AWS services like AWS Lambda, Amazon Kinesis, Amazon S3, and more. It is a complete rework, built to improve reliability, performance, and security. We invite your feedback! The SDK is built on top of a modified Paho MQTT Python client library. Developers can choose from two types of connections to connect to AWS IoT. For MQTT over TLS (port 8883 and port 443), a valid certificate and a private key are required for authentication. ... -
25
AWS Agent Skills
AWS Skills for Agents
...Instead of streaming giant documentation sets or relying on episodic web search, this project compresses AWS best practices, usage patterns, edge cases, and real-world engineering guides into pre-structured skill definitions that are token-efficient and tailored for reasoning. The skills cover critical AWS services such as IAM, Lambda, DynamoDB, S3, API Gateway, EKS, and many more, letting agents offer actionable advice on infrastructure as code, debugging, security configurations, and architectural workflows. Skills are kept up to date with weekly documentation checks, ensuring they reflect current AWS patterns and service changes.
