Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Problem when fetching from variable

Help
costas
2006-05-05
2013-05-30
  • costas
    costas
    2006-05-05

    I have the weirdest problem with snoopy. I have writen a script for fetching data from allmusic.com for another sourceforge project , zina. It worked like a charm until recently where this is what happens. In a part of the code i call some pages using a variable like so   if($snoopy->fetch("$url"))
    which used to work just fine. Now it brings me back the wrong page. when i print the url and paste it on a browser it takes me to the right one. when i hard code the url and not use a variable it works correctly. I am busting my head all mornign trying to figure out what happens and i am in a total loss. Anyone from the snoopy team have any idea what might be happening? I can send the script to anyone who thinks he can and wants to help out. Thanks in advance.

     
    • McDope
      McDope
      2006-05-05

      try to do it like this: if($snoopy->fetch("{$url}"))

      in many scripts that use snoopy i saw this writing and i use it so in my scripts, too. it works fine for me :)

       
    • costas
      costas
      2006-05-05

      Thank you very much for your prompt answer. I am afraid that it didn't work though. :-( The thing is that in the script i am calling pages many times with the standard way with no problem whatsoever. Even in that part of the code it worked for at least two months like it was. Maybe some of the fetched data is messing snoopy up badly. I think that it's something that the project admins would like to look at. Anyway, thanks again for your time.

       
    • costas
      costas
      2006-05-09

      Anybody from the snoopy admins would like to take a look at this bug? It's really furstrating. I have a loop like this 
      for ($i=0; $i<$count-1;$i++)
      { if($snoopy->fetch("http://www.allmusic.com$album_link[$i]"))

      Where count is the number of an artist's albums.
      When you echo "http://www.allmusic.com$album_link[$i]"; you get perfectly valid adresses like
      http://www.allmusic.com/cg/amg.dll?p=amg&sql=10:7vcyxd0bjolk
      http://www.allmusic.com/cg/amg.dll?p=amg&sql=10:rtouakok5m3c
      http://www.allmusic.com/cg/amg.dll?p=amg&sql=10:y1uw6j487190
      http://www.allmusic.com/cg/amg.dll?p=amg&sql=10:1sdsa9ugw23g
      for instance. When you substitute in snoopy->fetch the url with the $album_link[$i] with any of the echoed ones it works great. Please any of the admins have any, any idea what might be up??

       
      • McDope
        McDope
        2006-05-09

        i just got an idea, try to do the url in an seperate string like this:
        $myurl = "http://www.allmusic.com".$album_link[$i];
        and then give snoopy the string for fetching.

        maybe this can be an workaround...

         
    • costas
      costas
      2006-05-10

      Thanks again, but i have tried that already and it didn't work. In fact i have tried all the ways one could think of to pass that variable. Some that coe to mind right now are...
      (1)
      if($snoopy->fetch("http://www.allmusic.com$album_link[$i]"))
      (2)
      $server="http://www.allmusic.com"
      if($snoopy->fetch($server$album_link[$i]))
      (3)
      $server="http://www.allmusic.com"
      if($snoopy->fetch($server.$album_link[$i]))
      (4)
      $server="http://www.allmusic.com"
      if($snoopy->fetch("$server.$album_link[$i]"))
      (5)
      $server="http://www.allmusic.com"
      $url=$album_link[$i];
      if($snoopy->fetch($server$url))
      and all the above ways with the two variable way. i have tried every and all combinations i can think of. nothing worked. i am convinced that it has to do with snoopy and not with anything else. But again, i thank you for your time..

       
    • Shaun
      Shaun
      2007-09-15

      i have this exact same problem, i'm able to loop through a search form and receive search results but when i pass the link as a variable i don't get any return data, though it looks as if it is a limitation on the length of the link because i am getting an error page back with some of the GET information but it all needs to be there to work correctly and it isn't all there, but if i print the dynamic link out and quote it like: $snoopy->fetch("entire link") it works but if id use the variable: $snoopy->fetch($link) then i get the error page saying the information was passed but not all of it apparently because the page didn't show up the error page did.

      i also found that no combination of text to variable worked either, and i even tried $snoopy->submit() without a blank variable for the form variables and that wouldn't even work.

      this is a great class and is the basis for 90% of this project i'm working on but i'm spending an ungodly amount of time trying to figure out what i'm screwing up when i'm doing everything right, it's fetch() having trouble handling variables or maybe just big variables like >255 char.

       
      • sharp
        sharp
        2007-09-17

        Hello... I`m a very experienced user with Snoopy and I might be able to help you out. Yes, it is true that Snoopy fails now and then, due to bad written code. I`ve developed a new generation of crawler class called "Leech". If you wish, I can give you that class for your personal use, but I will not allow usage of it in comercial projects. In other words, if you wish to sell the project in which you use the class, the license will not be available anymore. Cheers! Oh, just so you know ... soon there will be a website for this new class.

         
    • Try changing the value of $read_timeout. It seems that the default 0 value (disallow timeouts) doesn't work very well. Instead use a large value.