crawel multithreads pages

  • hi i need some advice to start using this amazing script here is my project :
    i want to crawl a specific content of the threads links and add each content of that thread in separate data files (sqlite)
    i will put my initial code but i need some help what class the i need to force the cracker to read  specific content of the thread links .

  • Hi!

    What do you mean with "force the cracker to read specific content of the thread links"?
    What cracker? ;)

    Could you post your script to understand what you mean?


  • I'm sorry i was not able to modify my post  yesterday to correct the mistaken word cracker i meant crawler :)  here is the initial code but still need a lot of work :

    header('Content-type: text/html; charset=utf-8');
       include_once('simple_html_dom.php'); //PHP Simple HTML DOM Parser library 
       $html = file_get_html(''); 
    // Find all article blocks
    foreach($html->find('div.postbody') as $article) {
        $item = $article->find('div.content', 0)->innertext;
        $articles = $item;

    try {

    $db = new SQLite3('db/mysqlitedb.db');

    $db->exec('CREATE TABLE gg (bar STRING)');
    $db->exec("INSERT INTO gg (bar) VALUES ('this test')"); // i want  to tell $db to store $article values

    $result = $db->query('SELECT bar FROM gg');


        /* Close connections to Database */
        $db = NULL;

    catch( PDOException $e )

        /* Print Error-Messages */
        echo $e->getMessage();

    by this code i managed to parse just one thread content page and view it but still try to figure out how to use phpcrawl to read specific url and scan its threads links  and parse it's html content  and store it in it's own sqlite3  file data base i mean i want away to generate dynamically sqlite database file for each page thread  after the  phpcrawl read it 

  • Okay, you want to store content-parts of a/the website and all its subpages in a sqlite database file (like everything in $articles in your example?)

    If so i will post an example-script tomorrow (or the next days), don't have much time right now.

    Best Regards!

  • yes i want to store the content ($articles)  in sqlite3_db file
    and retrieve it  from the same database to view it in the browser ..

  • any update on this issue?



Cancel   Add attachments