crawler

A toolchain for bringing web2 to web3. IPFS daemon should be launched. Download enwiki-latest-all-titles to crawler root dir. Basically, there are two main functions provided by crawler tool. The first one is to parse wiki titles and submit links between keywords and wiki pages. Also, crawler has separate command upload-duras-to-ipfs to upload files to local IPFS node. All DURAs are collected under single root unixfs directory.

Features

Note: Requires Go 1.12+
IPFS daemon should be launched
Download enwiki-latest-all-titles to crawler root dir
Basically, there are two main functions provided by crawler tool
Parse wiki titles and submit links between keywords and wiki pages
Upload duras to IPFS

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow crawler

crawler Web Site

Other Useful Business Software

Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.

Try It Free

Rate This Project

User Reviews

Be the first to post a review of crawler!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Related Categories

Go File Sharing Software

Registered

2023-07-27

Similar Business Software

Proton Drive

Proton Drive is the all-in-one workspace for storing sensitive data and collaborating with your teams, clients, and partners. Collaborate securely without compromising control: Share client files, contracts, and sensitive business documents with full end-to-end encryption. You control who...

See Software
Cerberus FTP Server

Cerberus FTP Server provides a secure and reliable file transfer solution for the demanding IT professional in any industry. Supporting SFTP and SCP, FTP/S, and HTTP/S, Cerberus is able to authenticate against Active Directory and LDAP, run as a Windows service, has native x64 support, includes...

See Software
MASV

MASV Inc. is a secure cloud software company designed to quickly transfer heavy media files worldwide to meet fast-paced production schedules. Global media organizations rely on MASV to automatically deliver their large files without any restrictions, allowing them to concentrate on their next...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software
pCloud Business

pCloud is a cloud-based digital asset management platform and cloud storage that provides access to all your digital content including images, video, audio, docs, and more- anytime, anywhere, on any device. Keep all of your important files safe and centralized in one place. You can share with...

See Software
Diplomat Managed File Transfer

Diplomat MFT by Coviant Software is a secure, reliable managed file transfer solution designed to simplify and automate SFTP, FTPS, and HTTPS file transfers. Built for seamless integration, Diplomat MFT works across major cloud storage platforms, including AWS S3, Azure Blob, Google Cloud,...

See Software