vB-mTurk-Scraper-1.1 download

vB-mTurk-Scraper
Can scrape vBulletin forums for links to mTurk HITS

Written in: Python 2.* (Not 3 Compatible OR Tested)
Requires: BeautifulSoup and requests

To install BeautifulSoup run: pip install beautifulsoup4 OR: easy_install beautifulsoup4

To install requests run: pip install requests OR: easy_install requests

Information required to run program:

HITS Thread Number (Changes Daily: 5 digit number found in the thread URL)
Forum URL
Page To Start From
Number Of Pages To Scrape

This is a very very simple command-line script. Simply run python vB-mTurk-Scraper.py and it will guide you through setting the forum you want to scrape, entering todays thread number, which page you want to start from, and how many pages you want to go through.

Note: When entering the address to the forum, enter only the domain, ex: forum.com

It outputs all HITS to the html file mturklinks.html and always overwrites it self.

Read more @: http://git.io/vqpHS

Project Samples

vB-mTurk-Scraper running on Xubuntu 15.04

Project Activity

See All Activity >

Follow vB-mTurk-Scraper-1.1

vB-mTurk-Scraper-1.1 Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

Build gen AI apps with an all-in-one modern database: MongoDB Atlas

MongoDB Atlas provides built-in vector search and a flexible document model so developers can build, scale, and run gen AI apps without stitching together multiple databases. From LLM integration to semantic search, Atlas simplifies your AI architecture—and it’s free to get started.

Start Free

Rate This Project