crawler

A toolchain for bringing web2 to web3. IPFS daemon should be launched. Download enwiki-latest-all-titles to crawler root dir. Basically, there are two main functions provided by crawler tool. The first one is to parse wiki titles and submit links between keywords and wiki pages. Also, crawler has separate command upload-duras-to-ipfs to upload files to local IPFS node. All DURAs are collected under single root unixfs directory.

Features

Note: Requires Go 1.12+
IPFS daemon should be launched
Download enwiki-latest-all-titles to crawler root dir
Basically, there are two main functions provided by crawler tool
Parse wiki titles and submit links between keywords and wiki pages
Upload duras to IPFS

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow crawler

crawler Web Site

Other Useful Business Software

Ship Agents Faster

Transform your applications and workflows into powerful agentic systems at global scale.

Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.

Get Started Free

Rate This Project

User Reviews

Be the first to post a review of crawler!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Related Categories

Go File Sharing Software

Registered

2023-07-27

Similar Business Software

Cerberus FTP Server

Cerberus FTP Server provides a secure and reliable file transfer solution for the demanding IT professional in any industry. Supporting SFTP and SCP, FTP/S, and HTTP/S, Cerberus is able to authenticate against Active Directory and LDAP, run as a Windows service, has native x64 support, includes...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software
pCloud Business

pCloud is a cloud-based digital asset management platform and cloud storage that provides access to all your digital content including images, video, audio, docs, and more- anytime, anywhere, on any device. Keep all of your important files safe and centralized in one place. You can share with...

See Software
Diplomat Managed File Transfer

Diplomat MFT by Coviant Software is a secure, reliable managed file transfer solution designed to simplify and automate SFTP, FTPS, and HTTPS file transfers. Built for seamless integration, Diplomat MFT works across major cloud storage platforms, including AWS S3, Azure Blob, Google Cloud,...

See Software
ONLYOFFICE Docs

ONLYOFFICE is an open-source project that offers cloud-based and self-hosted solutions for business of all sizes. The key product is ONLYOFFICE Docs, a secure office suite that seamlessly integrates into the most popular platforms, e.g. Odoo, Alfresco, Confluence, Pipedrive, Redmine, SuiteCRM...

See Software
Filecamp

Filecamp is a cloud-based Digital Asset Management system that allows marketing professionals to organize, share, tag, and distribute their digital assets. All from their own custom-branded, password-protected, easy-to-use, space in the cloud. Filecamp is used by over 1,600 brands from more...

See Software