vB-mTurk-Scraper
Can scrape vBulletin forums for links to mTurk HITS

Written in: Python 2.* (Not 3 Compatible OR Tested)
Requires: BeautifulSoup and requests

To install BeautifulSoup run: pip install beautifulsoup4 OR: easy_install beautifulsoup4

To install requests run: pip install requests OR: easy_install requests

Information required to run program:

HITS Thread Number (Changes Daily: 5 digit number found in the thread URL)
Forum URL
Page To Start From
Number Of Pages To Scrape

This is a very very simple command-line script. Simply run python vB-mTurk-Scraper.py and it will guide you through setting the forum you want to scrape, entering todays thread number, which page you want to start from, and how many pages you want to go through.

Note: When entering the address to the forum, enter only the domain, ex: forum.com

It outputs all HITS to the html file mturklinks.html and always overwrites it self.

Read more @: http://git.io/vqpHS

Project Samples

vB-mTurk-Scraper running on Xubuntu 15.04

Project Activity

See All Activity >

Follow vB-mTurk-Scraper-1.1

vB-mTurk-Scraper-1.1 Web Site

Other Useful Business Software
Build Agents and Models on One Platform Icon
Build Agents and Models on One Platform

Everything you need to build production-ready agents and models. Access 200+ Google and third-party AI models and tools.

Gemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
Start Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of vB-mTurk-Scraper-1.1!

Additional Project Details

Registered

2015-07-12
Report inappropriate content