crawler

A toolchain for bringing web2 to web3. IPFS daemon should be launched. Download enwiki-latest-all-titles to crawler root dir. Basically, there are two main functions provided by crawler tool. The first one is to parse wiki titles and submit links between keywords and wiki pages. Also, crawler has separate command upload-duras-to-ipfs to upload files to local IPFS node. All DURAs are collected under single root unixfs directory.

Features

Note: Requires Go 1.12+
IPFS daemon should be launched
Download enwiki-latest-all-titles to crawler root dir
Basically, there are two main functions provided by crawler tool
Parse wiki titles and submit links between keywords and wiki pages
Upload duras to IPFS

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow crawler

crawler Web Site

Other Useful Business Software

$300 Free Credits to Build on Google Cloud

New customers can spin up VMs, build with AI, and query data at no cost.

Put your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.

Start Free

Rate This Project

User Reviews

Be the first to post a review of crawler!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Related Categories

Go File Sharing Software

Registered

2023-07-27

Similar Business Software

Proton Drive

Proton Drive is the all-in-one workspace for storing sensitive data and collaborating with your teams, clients, and partners. Collaborate securely without compromising control: Share client files, contracts, and sensitive business documents with full end-to-end encryption. You control who...

See Software
Cerberus FTP Server

Cerberus FTP Server provides a secure and reliable file transfer solution for the demanding IT professional in any industry. Supporting SFTP and SCP, FTP/S, and HTTP/S, Cerberus is able to authenticate against Active Directory and LDAP, run as a Windows service, has native x64 support, includes...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software
pCloud Business

pCloud is a cloud-based digital asset management platform and cloud storage that provides access to all your digital content including images, video, audio, docs, and more- anytime, anywhere, on any device. Keep all of your important files safe and centralized in one place. You can share with...

See Software
Diplomat Managed File Transfer

Diplomat MFT by Coviant Software is a secure, reliable managed file transfer solution designed to simplify and automate SFTP, FTPS, and HTTPS file transfers. Built for seamless integration, Diplomat MFT works across major cloud storage platforms, including AWS S3, Azure Blob, Google Cloud,...

See Software
ONLYOFFICE Docs

ONLYOFFICE is an open-source project that offers cloud-based and self-hosted solutions for business of all sizes. The key product is ONLYOFFICE Docs, a secure office suite that seamlessly integrates into the most popular platforms, e.g. Odoo, Alfresco, Confluence, Pipedrive, Redmine, SuiteCRM...

See Software