Captcha request -- how to handle?

Help
2010-11-26
2013-05-13
  • Peter Loron
    Peter Loron
    2010-11-26

    I'm trying to use esniper 2.24 and am getting the dreaded captcha security request, which prevents esniper from working. Any workaround?

     
  • Scott Nicol
    Scott Nicol
    2010-11-27

    There are no shortcuts around this.  You need to use eBay with a web browser and answer the captchas until eBay stops asking.  Might take a week or two.

    Try to figure out why your account got flagged, and don't do it.  See the captcha page for a few ideas.

     
  • Android
    Android
    2010-11-27

    The captcha only happens on bidding history page.  The main listing there is no captcha, and I've never seen a captcha on a bid.  Why do we care about the bid history?  The only thing we should care about is the current price and that can easily be found on the main page of the listing.

    eBay is becoming more and more difficult with the captchas all the time.  It really doesn't matter what you do there's a good chance you're going to get one.  I've been using esniper for at least two years and had never seen a captcha on eBay.  Since August I've had three and I'm not changing my bidding or purchasing habits.  I rarely use esniper these days because I don't trust it.

    I don't know why there is such a push to process the raw HTML and search for information.  Why not process the page(aka generate the actual resulting text), pull the data and go from there?  Every time eBay changes how they generate the item's page esniper breaks horribly for a week or two.  It took over a year to get the watch list working again since the last change from eBay.  Screen scraping is not a new idea, why does esniper make it so difficult?

    I don't want to be too harsh or to criticize those who are providing a solution, but isn't this a case of working hard instead of working smart?

    And saying "don't do that" doesn't work.  This is a case of eBay changing how they do things, and if esniper doesn't change then it's going to die.  Look at how many defunct sniper apps are out there…

     
  • Scott Nicol
    Scott Nicol
    2010-11-27

    esniper used the item description page for the first 5 years of its life, but switched to bid history more than 8 years ago (see esniper.c rev 1.4) because the bid history seemed to change format less often than the item description.  As for what info esniper needs to know, you are missing the most obvious thing - the time remaining!

    The first thing you do when scraping web pages is to get the raw html.  You can then, if you like, render it and grab the visual parts.  But other than for handling tricks like decoding data generated by javascript, rendering the page is a wasted effort.  The data is already available to you in the html, and the tags and variables make it easier to find the correct data.  If the raw html has changed enough to break esniper, chances are the visual presentation has also changed enough too.

    esniper has been around longer than ebay (I originally wrote it to snipe onsale).  It has changed a lot in 15 years.  However, there are only two developers, and both of us have lives outside of esniper.  This means there is a built-in resistance to make wholesale changes to esniper unless there is a real need for it.

    If you want to take esniper in a different direction, feel free to dig in.  If we like your work, we'll patch it into the mainline.  If we really like your work, you'll be invited to join the team.  If we don't like your work but you think it's awesome, fork!  I've got no problem with that.

     
  • Tedd
    Tedd
    2010-12-02

    In understand the developers point of view concerning the effort they put into to such a free project, but I have a problem with always getting captcha warnings, too. First I used esniper for a long time very successful on ebay.de but now I always get an error (no captcha). I tried to send in the bug report, but I get a failure on souceforge.net. If I try to use ebay.com I always get an captcha error with esniper. But I never got a captcha if I use the web browser. Might it be a parser error, which let esniper "think" that there is a captcha? I already looked into the source code, but I am no C guru… I cannot find a problem with it, after years of work of the two developers… Could I send you a bug report by mail?

     
  • Scott Nicol
    Scott Nicol
    2010-12-03

    Sure, just email esniper@users.sourceforge.net

     
  • ocman
    ocman
    2010-12-04

    Been a long time user of esniper, and appreciate the work that has been put into it.  When it does work, it works well and I like being able to run my own management cgi scripts to configure it via web.  Some comments if I may…

    It seems like basing esniper off the bid history page is not working out well, considering that the captcha system basically cripples esniper and makes it unreliable and unusable.  Why not base it off the item page itself?  I think all the information necessary is readily available on that page, and I've never seen a captcha occur for an item page.  Otherwise, you might as well consider closing down the project, because captcha is here to stay.

    Captcha + bid history page = perpetually broken esniper

     
  • Scott Nicol
    Scott Nicol
    2010-12-04

    eBay doesn't currently require login to see the item page, so you shouldn't get a captcha.  They do require login for bid history and bidding.  Bid history is easier to parse, but if it becoming too much of a pain we'll just have to retarget.

    bid history format is consistent across all auctions.  item description page format can be different for different item types.  I don't know if eBay does this anymore, but at the time esniper was switched to bid history, eBay was creating boutique sites (like motors) that had their own page format.

     
  • Tedd
    Tedd
    2010-12-05

    May be I am totally wrong, but captchas never appear during a bidding process, but only when starting a group and hence esniper tries to log into the bid history page. Captchas are recognised after the whole page was fetched with curl. I assume that many users use esniper in conjunction with es-f. What about a command line parameter which does not just log the occurred "captcha-event" but saves the whole page (which is already loaded for parsing purposes). This would open the opportunity to open the page and solve the captcha manually within a browser from the same machine and ip esniper runs on… Maybe es-f would integrate such an output within their frontend…

     
  • Scott Nicol
    Scott Nicol
    2010-12-12

    Captcha requests can also happen during bidding. if esniper was changed to use the description page, you would see this. Many other sniper programs use the description page and also have problems with captchas.

     
  • Scott Nicol
    Scott Nicol
    2010-12-12

    Btw, ebay has had captcha for at least 5 years. I have never been asked to respond to a captcha for my main ebay account, only on a"test" account, and I really had to try to get ebay to mark my account for captcha. Maybe ebay is getting more aggressive?

     
  • Tedd
    Tedd
    2010-12-12

    That's is the "funny" point. I use the same account for esniper and the webbrowser. I know that this might cause problems but I never had problems for at least a year; since beginning using esniper. I never never got a captcha using a webbrowser, but only with esniper which runs on a server with static ip. Hence I installed xampp on my laptop together with esniper. Result: captacha. Other Ebay account, result: captacha. Might it by a problem of the http header and the transmitted browser agent send by curl?

     
  • Android
    Android
    2010-12-12

    I would be very surprised if eBay did a captcha on the main page during the end of an auction.  Maybe if you were reloading the page ten times a second it would trigger something, but even without esniper running I still snipe by hand.  I know I'm not the only one who does it that way, and if a captcha came up and blocked my bid in the last 15-20 seconds of an auction I'd be pissed (and as a seller if I found out eBay was stopping buyers from bidding on my auctions that I'm paying for…).

    I just don't see them doing it, as it would screw around with the business of selling.  Then again I've been watching what they've been doing to sellers with the new policies and I can't believe how many sellers they've managed to "run off" the site.

    Why not try it?  The worse that can happen is we end up where we are now.

     
  • Scott Nicol
    Scott Nicol
    2010-12-12

    Agent is not the default, but the one that esiper uses is way out of date (mozilla 4.x on linux 2.2). Hey, at least it isn't mozilla 2 on hp-ux!

    I'll change the default to something more current, maybe IE on vista. I'll also add a configuration option to set it to whatever you want.

     
  • Scott Nicol
    Scott Nicol
    2010-12-12

    Ebay is stupid enough to do a captcha when bidding. Other snipers that only use the description page still run into captchas.

    Lack of time is a major factor in not making the change to the description page. The major amount of time isn't writing the code, it is testing. If I get a couple of hours I'll try to do it, but I'll stick it on a branch until I'm confident it is ok.