vB-mTurk-Scraper-1.1 download

vB-mTurk-Scraper
Can scrape vBulletin forums for links to mTurk HITS

Written in: Python 2.* (Not 3 Compatible OR Tested)
Requires: BeautifulSoup and requests

To install BeautifulSoup run: pip install beautifulsoup4 OR: easy_install beautifulsoup4

To install requests run: pip install requests OR: easy_install requests

Information required to run program:

HITS Thread Number (Changes Daily: 5 digit number found in the thread URL)
Forum URL
Page To Start From
Number Of Pages To Scrape

This is a very very simple command-line script. Simply run python vB-mTurk-Scraper.py and it will guide you through setting the forum you want to scrape, entering todays thread number, which page you want to start from, and how many pages you want to go through.

Note: When entering the address to the forum, enter only the domain, ex: forum.com

It outputs all HITS to the html file mturklinks.html and always overwrites it self.

Read more @: http://git.io/vqpHS

Project Samples

vB-mTurk-Scraper running on Xubuntu 15.04

Project Activity

See All Activity >

Follow vB-mTurk-Scraper-1.1

vB-mTurk-Scraper-1.1 Web Site

Other Useful Business Software

Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.

Try free now

Rate This Project