Crawler Code
Status: Alpha
Brought to you by:
fccdotgov
Menu
▾
▴
Tree [r1] / History
| File | Date | Author | Commit |
|---|---|---|---|
| includes | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| README.txt | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| browse.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| config.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| crawl.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| create-tables.sql | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| export.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| query.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| sitemap.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
| stats.php | 2010-09-07 | fccdotgov | [r1] Initial Commit |
Read Me
TO USE: 1. Edit config.PHP with appropriate database and domain information 2. (for now) in phpMyAdmin insert the seed URL into the urls table. * URL should be www. * URL should have a trailing slash * (for now) May also want to set clicks to '0' to avoid problems 3. Open crawler.php 4. (optional) open stats.php to watch progress TIPS: Changes to php.ini 1. Increase memory limit (1GB) 2. Remove execution time limit Changes to mysql.ini * Increased max query size (to avoid "mysql went away" error) Additional documentation (source code) in (/source)
