#694 Automated bug report (Page "Black Friday")

[Brainbug02]

Automated esniper bug report.
esniper version 2.31.0
libcurl/7.43.0 OpenSSL/1.0.2d zlib/1.2.8 libidn/1.28 librtmp/2.3
Error encountered in function ebayLogin in auction.c line 489
auction = 161897120811, price = 25, remain = 0
latency = 0, result = -1, error = 19
buf = 0x7f044ec40010, size = 175032, read = 0x7f044ec40010
time = 1448484217, offset = 0
pagename = "Black Friday Deals Now On!", pageid = "(null)", srcid = "(null)"
specified options or config values:
1 x username(u) =
1 x password() =
1 x seconds(s) = 5
1 x quantity(q) = 1
1 x (f) = "kreisschneider"
1 x reduce() = 1
1 x bid() = 1
1 x debug(d) = 1
1 x batch(b) = 0
1 x logdir(l) = "/home/me/esniper/log"
unknown pageinfo

[Brainbug02]

I've started the same auction file yesterday without any problems. However today some other auctions failed with the same error.
I'm able to log in manually without any notice whatsoever.

[Bodo]

The attached page seems to show special offers after successful login. I guess it indicates successful login because it shows"Hi Lars!" in the top line.
It seems difficult to check for successful login with this change. (A detailed analysis is necessary.)
Probably it is a temporary change (because of "black friday").
As a workaround: Stop bidding on auctions for a few days. Maybe try again on Saturday.

[ticketack]

Hi,

since this mornig, I have the same problem like Brainbug02. Same error on different machines

Strange

[Bodo]

Let's see if this is a temporary change. If not we need a modified login method.
Ebay used to present certain pages like "My Ebay Summary" after successful login. Currently esniper gets a page that is difficult to recognize as a successful login.

[Brainbug02]

I've tried to change the server to ebay.de where no black Friday page is displayed - without any changes. The dumped html file was the regular ebay landing page (after login).

[ticketack]

I have also recompiled the source and changed login page to#define DEFAULT_LOGIN_HOST "signin.ebay.de", which redirects to the login page.

The result is just a german pagename in error output log:
pagename = "Elektronik, Autos, Mode, Sammlerst ?cke, Gutscheine und mehr Online-Shopping | eBay", pageid = "(null)", srcid = "(null)"

So, something went realy wrong during login

[ticketack]

The login itself seems to be okay. In the detailed log files I have found this line.

2015-11-26 16:55:14.149157 https://signin.ebay.de/ws/eBayISAPI.dll?SignInWelcome&userid=MYLOGINNAME&pass=**&keepMeSignInOption=1

MYLOGINNAME was replaced by my real login name.

I have copied this to the addressline of my browser and replaced the five stars (*) with my real password, The "#" in my password I have substituted by "%23" and this works fine fine for signin.ebay.de and for signin.ebay.com as well.

I will check the logs later, I have to leave now. The log for one failed bid/login is > 22k lines - wow.

[Bodo]

I agree that the login is successful, but esniper does not recognize the resulting page. Some years ago Ebay's HTML pages used to have machine-readable page IDs in the Javascript code, but currently we have to check the page title or even check other strings in the document text.

After login esniper accepts a page title starting with "Electronics" (English page from default server) but not with "Elektronik" (German server) or any other languages. It also accepts several spelling variants of the myEbay page.
To accept the German page, a check for the corresponding text has to be added to the source code. The same applies to other languages.

BTW: In order to use different servers it is not necessary to change the default host names in the source code, you can use config file options "loginHost" etc at run time.

My idea for an improved login mechanism is to use the existing login URLs and check for known page titles that indicate a login failure but ignore other pages. After this try to get the myEbay page and check its page title. With manual tests I get a login page instead of the myEbay page when I'm not logged in.

[ticketack]

Bodo, you are right. esniper expects a page with a proper string.

I have modified the appropriate "if" routine in auction.c, line 474, in that way, that it will pass always, independent if login was correctly or not.

line 474:   )||(1==1)) //to avoid login failure bug 694  - not a fine way

It works.

Good night

[Brainbug02]

One thing that I noticed is that nearly all ebay pages contain a JS line of the following format

{GH_config={"siteId":"77","geoLang":"[]",sin:1,id:'USER',fn:'FIRSTNAME',pageId:2054900,ct:0};GH.init();}

were USER and FIRSTNAME are the user's username and firstname respectively. Other config strings can be included as well.

When you are not logged in, the line looks like this

{GH_config={"siteId":"77","geoLang":"[]","largeDoodle":"http://ir.ebaystatic.com/pictures/aw/uk/holiday/Doodle2015/DE_DOODLE/7375_DE_Retail_Superweekend_Doodle_245x400_BlackFriday.png",sin:0,id:'',fn:'',pageId:2050601,ct:0};GH.init();}

So by checking the value of id and fn one could check if the user is logged in. Maybe this method is more reliable than checking the page titles.

However I haven't checked yet if this kind of check can be done in auction.c.

[ticketack]

Yeah, this seems to be a good method to verify succeed of login.
I am not familiar with the code, but I believe a simple string compare function of

,sin:1,id:'USERNAME',fn:

is no problem.
The way I had made esniper working again was a dirty hack to bring the tool back to life for coming weekend.

[Brainbug02]

I've changed the function to check for the id. This is how it could look like:

Index: auction.c
===================================================================
RCS file: /cvsroot/esniper/esniper/auction.c,v
retrieving revision 1.131
diff -u -r1.131 auction.c
--- auction.c   31 Jul 2014 15:44:31 -0000      1.131
+++ auction.c   27 Nov 2015 12:09:58 -0000
@@ -418,9 +418,10 @@
        memBuf_t *mp = NULL;
        size_t urlLen;
        char *url, *logUrl;
-       pageInfo_t *pp;
        int ret = 0;
        char *password;
+       int logged_in = 0;
+       const char *line;

        /* negative value forces login */
        if (loginTime > 0) {
@@ -459,45 +460,23 @@
        if (!mp)
                return httpError(aip);

-       if ((pp = getPageInfo(mp))) {
-               log(("ebayLogin(): pagename = \"%s\", pageid = \"%s\", srcid = \"%s\"", nullStr(pp->pageName), nullStr(pp->pageId), nullStr(pp->srcId)));
-               /*
-                * Pagename is usually MyeBaySummary, but it seems as though
-                * it can be any MyeBay page, and eBay is not consistent with
-                * naming of MyeBay pages (MyeBay, MyEbay, myebay, ...) so
-                * esniper must use strncasecmp().
-                */
-               if ((pp->srcId && !strcmp(pp->srcId, "SignInAlertSupressor"))||
-                   (pp->pageName &&
-                       (!strncasecmp(pp->pageName, "MyeBay", 6) ||
-                        !strncasecmp(pp->pageName, "My eBay", 7) ||
-                        !strncasecmp(pp->pageName, "Watch list", 10) ||
-                        !strncasecmp(pp->pageName, "Purchase History", 16) ||
-                        !strncasecmp(pp->pageName, "Electronics", 11))
-                   ))
-                       loginTime = time(NULL);
-               else if (pp->pageName &&
-                               (!strcmp(pp->pageName, "Welcome to eBay") ||
-                                !strcmp(pp->pageName, "Welcome to eBay - Sign in - Error") ||
-                                !strcmp(pp->pageName, "Welcome to eBay - Error")))
-                       ret = auctionError(aip, ae_badpass, NULL);
-               else if (pp->pageName && !strcmp(pp->pageName, "PageSignIn"))
-                       ret = auctionError(aip, ae_login, NULL);
-               else if (pp->pageName && !strcmp(pp->pageName, "Reset your password"))
-                       ret = auctionError(aip, ae_manualaction, NULL);
-               else if (pp->srcId && !strcmp(pp->srcId, "Captcha.xsl"))
-                       ret = auctionError(aip, ae_captcha, NULL);
-               else {
-                       ret = auctionError(aip, ae_login, NULL);
-                       bugReport("ebayLogin", __FILE__, __LINE__, aip, mp, optiontab, "unknown pageinfo");
-               }
-       } else {
-               log(("ebayLogin(): pageinfo is NULL\n"));
+       memReset(mp);
+       while ((line = getNonTag(mp)) && (!logged_in)) {
+               char *tmp;
+
+               if (tmp = strstr(line, ",id:\'"))
+                       if (*(tmp+6) != '\'')   // username is not empty
+                               logged_in = 1;
+       }
+
+       if (!logged_in)
+       {
+               bugReport("ebayLogin", __FILE__, __LINE__, aip, mp, optiontab, "empty user id");
+               log(("ebayLogin(): empty user id\n"));
                ret = auctionError(aip, ae_login, NULL);
-               bugReport("ebayLogin", __FILE__, __LINE__, aip, mp, optiontab, "pageinfo is NULL");
        }
+
        freeMembuf(mp);
-       freePageInfo(pp);
        return ret;
 }

This patch worked for me using ebay.com. However as I've tried some wrong passwords ebay forces me to solve captchas ;-) So I can't try ebay.de or ebay.co.uk by now...

[Bodo]

I would prefer to add the search for the id string to getPageInfo() and return the login flag as part of the page info structure. I suggest to check for some more text because the short search string ",id:'" looks a bit dangerous to me.

[Robert M. Münch]

Can't we x-check / use an ebay cookie? IIRC it's possible to auto-login when accessing an ebay page.

[Bodo]

I would be interested in a detailed description.
AFAIK ebay login is not permanent. The server asks for the password from time to time. (Don't know details about timeout or conditions.) That's why esniper has a mechanism to renew the login regularly and to make sure to be logged in some time (10min?) before bidding.

[Robert M. Münch]

I can't remember if this is in the right direction: http://stackoverflow.com/questions/24921029/are-ebay-user-tokens-permanent-fetch-user-token-api

IIRC these tokens can be used to work against the API.

And here is one more, someone using curl to do it:

http://stackoverflow.com/questions/16611362/curl-login-into-ebay-co-uk

[Brainbug02]

I suggest to check for some more text because the short search string ",id:'" looks a bit dangerous to me.

It's dangerous, for sure ;-) However I've checked quite a few pages and ",id:'" was unique. One could extend the string to "sin:NUMBER,id:'',fn:''" which makes it even more unique but error-prone if ebay adds attributes inbetween.

In the end every mechanism can fail if ebay changes the page layout... ;-(

[Robert M. Münch]

This patch worked for me on ebay.de as well. Great stuff!!

But, can the repro of this project be moved to GitHub SF is just crap...

[Bodo]

...very detailed problem description about SF...
I tried to move from CVS to Git but did not yet finish to adapt the release script and instructions. We have a partially automated release mechanism that calls CVS commands. This needs to be changed for a transition to Git. (I prefer Git over CVS or SVN, but for me it's not worth the effort.)

[Christian Burger]

Hey Bodo, what if someone else would make the move for you and transfer the newly created project over to you? I see that there is not much to gain from the switch to GitHub for you if no one else steps up and shows that there is indeed interest from other parties.

If time is no issue, I would step up and volunteer to transfer the project to GitHub in the coming months and hand it over to you when I am done. If you are interessted, let's discuss the details at [feature-requests:#31].

[Rolf]

Christian, are you still interested to drive such a transfer to github/gitlab?

It looks to me as if nobody would stop you and a number of people would very much welcome it. But please don't underestimate the needed work. It isn't only to upload the git repo to github.com and call it done. The migration will take some time and effort. Please continue discussion in feature request #31 if you want to volunteer.

[Bodo]

I continue the discussion about Git at [feature-requests:#31]