Menu ▾ ▴

#797 fails to login, consistently, suddenly

v1.0 (example)
open
nobody
None
5
2021-01-02
2020-08-15
Phil Dibowitz
No

Usually when esniper fails to log in, I open a browser, logout and back in to handle captcha and then it works. But today esniper was running and on a refresh of auctions failed to login, and crashed. I've logged out/in in a browser several times now and esniper still can't login.
Attached is the bug file.

1 Attachments
esniper.15471.7.bug.html

Discussion

  • Phil Dibowitz

    Phil Dibowitz - 2020-08-15

    Detail report from console:
    paste this into "Detailed Description":
    Automated esniper bug report.
    esniper version 2.35.0
    libcurl/7.68.0 GnuTLS/3.6.14 zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.8.0 nghttp2/1.41.0 librtmp/2.3
    Error encountered in function ebayLogin in auction.c line 672
    auction = 233675128459, price = 151, remain = 0
    latency = 0, result = -1, error = 19
    buf = 0x55dcd69a47b0, size = 93310, read = 0x55dcd69a47b0
    time = 1597471242, offset = 0
    pagename = "Sign in or Register | eBay", pageid = "(null)", srcid = "(null)"
    specified options or config values:
    2 x username(u) =
    2 x password() =
    2 x seconds(s) = 5
    1 x quantity(q) = 999
    1 x (f) = ".esniper_auction"
    2 x bid() = 1
    2 x batch(b) = 1
    unknown pageinfo

     

  • Phil Dibowitz

    Phil Dibowitz - 2020-08-15

    I tried updating to the latest commit on legacy branch and now I get:

    Auction 233675128459: Cannot connect to URL /distil_r_captcha.html?requestId=8ae0850d-72ee-4270-91c4-1fd7b132870d&httpReferrer=%2Fws%2FeBayISAPI.dll%3FSignIn: URL using bad/illegal format or missing URL:
    Retrying...
    Auction 233675128459: Cannot connect to URL /distil_r_captcha.html?requestId=8e1d56db-edb0-47e3-9aff-10f1cd9aea4a&httpReferrer=%2Fws%2FeBayISAPI.dll%3FSignIn: URL using bad/illegal format or missing URL:
    Retrying...
    ^C

     

  • Michael S.

    Michael S. - 2020-08-15

    Remark:

    You get the

    Cannot connect to URL /distil_r_captcha.html?requestId=8e1d56db-edb0-47e3-9aff-10f1cd9aea4a&httpReferrer=%2Fws%2FeBayISAPI.dll%3FSignIn: URL using bad/illegal format or missing URL:

    message, because this is the brute force protection (captcha image) and you have'nt used the latest commit from the legacy branch. If you do this, you get

    .
    .
    findAttr cannot find <label for="userid"
    then upload and attach esniper.30717.1.bug.html and click submit.

    which means, that an unexpected page has been returned.

    esniper searches for special content within the sign in page, which is needed as post parameter of the html request.

     

  • Phil Dibowitz

    Phil Dibowitz - 2020-08-16

    esniper searches for special content within the sign in page, which is needed as post parameter
    of the html request.

    So... what's the way past that?

     

  • Michael S.

    Michael S. - 2020-08-16

    Currently I am working on the master branch, which is using the modern login mechanism. But at the moment it doesnt work at all.
    Please be patient ...

     

  • Phil Dibowitz

    Phil Dibowitz - 2020-08-17

    Ah! Sorry, I didn't understand that. I thought there was still something I was doing wrong.

    As someone who develops opensource software myself, I understand that this is something you do in your free time, for free, and I greatly appreciate the time and effort you've put into esniper.

    Best of luck with your work on the new login mechanism, and I look forward to the results!

     

  • Michael S.

    Michael S. - 2020-08-17

    Same problem with other sniping tools since last year:

    http://forum.jbidwatcher.com/forums/1/topics/6797?page=1

    They also identified the cookies DGIID DGUID DGZID DGZUID DGHID DGSID as reason for the failure (see https://sourceforge.net/p/esniper/bugs/783/). This cookies are generated by nkfytkqtoxtljvzbxhr.js.

    There are several js engines available, e.g. https://duktape.org and similar projects, which could be called from a C program. But normaly nkfytkqtoxtljvzbxhr.js runs within a browser and has access to the browser and DOM context. I dont know, if nkfytkqtoxtljvzbxhr.js generates a valid "fingerprint" to get the DG-cookies without this context.

     

  • NickT

    NickT - 2020-08-17

    It looks like ebay have finally killed the legacy route. I managed to get round the 783 bug (which I raised) by rolling back to 2.35.0, as suggested by John Cuzzourt. That, and checking out the legacy version now both fail looking for a captcha URL.

     

    Last edit: NickT 2020-08-17

  • Michael S.

    Michael S. - 2020-08-18

    duktape doesn't work. There are several imports needed, which are available in the browser environment, but not within duktape. I'm afraid, that we have to use a "monster" like v8.

     

  • Michael S.

    Michael S. - 2020-08-18

    Another possibility could be translating nkfytkqtoxtljvzbxhr.js into native C code.

    The online tools I tried, won't do that job.

     

  • Michael S.

    Michael S. - 2020-08-24

    I've installed nodejs for test purposes:

    $ curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -

    (see https://linuxhint.com/install_npm_debian/)

    $ node --version
v12.18.3
$ npm --version
6.14.6

    Package:

    $ less package.json
{
  "name": "ebay",
  "version": "1.0.0",
  "description": "ebay signin script",
  "main": "signin.js",
  "scripts": {
    "test": "node --check signin.js",
    "start": "node --trace-warnings signin.js"
  },
  "author": "msp",
  "license": "ISC",
  "dependencies": {
    "canvas": "^2.6.1",
    "express": "^4.17.1",
    "jsdom": "^16.4.0",
    "tough-cookie": "^4.0.0"
  },
  "devDependencies": {}
}

    Script:

    $ less signin.js
const url = "https://signin.ebay.com/ws/eBayISAPI.dll?SignIn";
const jsdom = require("jsdom");
const cookieJar = new jsdom.CookieJar();
const { JSDOM } = jsdom;
options = {
    runScripts: 'dangerously',
    resources: "usable"
}
JSDOM.fromURL(url, options, { cookieJar }).then(dom => {
//  console.log(dom.serialize());
    console.log("Loading " + url + " ok.")
});

var window = JSDOM.window
var document = JSDOM.document

class NAVIGATOR {
   language = "en"
   cpuClass = "unknown"
   platform = "unknown"
   doNotTrack = "unknown"
   plugins = { }
   maxTouchPoints = 0
   appName = "unknown"
}

class SCREEN {
   width = 1024
   height = 768
   availHeight = 768
   availWidth = 1024
   pixelDepth = 16
}

const screen = new SCREEN();
const navigator = new NAVIGATOR();

function cb(err, data) {
  console.log(data)
}

setTimeout(() => {
   console.log("Dumping CookieJar:")
   cookieJar.serialize(cb)
}, 2000)

    Install modules:

    $ npm install express --save
$ npm install jsdom
$ npm install canvas
$ npm install tough-cookie

    Start:

    $ npm start

> ebay@1.0.0 start MYPATH
> node --trace-warnings signin.js


> ebay@1.0.0 start MYPATH
> node --trace-warnings signin.js

Loading https://signin.ebay.com/ws/eBayISAPI.dll?SignIn ok.
Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
Error: Not implemented: navigation (except hash changes)
    at module.exports (MYPATH/node_modules/jsdom/lib/jsdom/browser/not-implemented.js:9:17)
    at navigateFetch (MYPATH/node_modules/jsdom/lib/jsdom/living/window/navigation.js:76:3)
    at exports.navigate (MYPATH/node_modules/jsdom/lib/jsdom/living/window/navigation.js:54:3)
    at LocationImpl._locationObjectNavigate (MYPATH/node_modules/jsdom/lib/jsdom/living/window/Location-impl.js:31:5)
    at LocationImpl.replace (MYPATH/node_modules/jsdom/lib/jsdom/living/window/Location-impl.js:231:10)
    at Location.replace (MYPATH/node_modules/jsdom/lib/jsdom/living/generated/Location.js:91:36)
    at XMLHttpRequest.r.onreadystatechange (https://signin.ebay.com/nkfytkqtoxtljvzb.js:1:25039)
    at XMLHttpRequest.<anonymous> (MYPATH/node_modules/jsdom/lib/jsdom/living/helpers/create-event-accessor.js:32:32)
    at innerInvokeEventListeners (MYPATH/node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:318:25)
    at invokeEventListeners (MYPATH/node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:274:3) undefined
Dumping CookieJar:
{
  version: 'tough-cookie@3.0.1',
  storeType: 'MemoryCookieStore',
  rejectPublicSuffixes: true,
  cookies: []
}

    The async. request XMLHttpRequest was called, but the implementation is missing.

    Without the options

    options = {
    runScripts: 'dangerously',
    resources: "usable"
}

    we get some cookies, but not the DG... cookies we need, because the script nkfytkqtoxtljvzbxhr.js will not be executed:

    {
  version: 'tough-cookie@3.0.1',
  storeType: 'MemoryCookieStore',
  rejectPublicSuffixes: true,
  cookies: [
    {
      key: 'dp1',
      value: 'bu1p/QEBfX0BAX19AQA**630643fe^pbf/%23200000000000000000000000000046125107e^tzo/1a45f43eb0e^bl/DE630643fe^',
      expires: '2022-08-24T15:30:06.000Z',
      domain: 'ebay.com',
      path: '/',
      hostOnly: false,
      creation: '2020-08-24T15:30:04.910Z',
      lastAccessed: '2020-08-24T15:30:04.910Z'
    },
    {
      key: 'nonsession',
      value: 'BAQAAAXIlFHjQAAaAADMABWElEH41MzE3MwDKACBjBkP+MjExNzQwN2ExNzQwYTc3MGM1MzViODRiZmZmOWE2OWEAywABX0PkBjE1awJNyiMXPpb6hKNk/cxTzqptRg**',
      expires: '2022-08-24T15:30:06.000Z',
      domain: 'ebay.com',
      path: '/',
      hostOnly: false,
      creation: '2020-08-24T15:30:04.914Z',
      lastAccessed: '2020-08-24T15:30:04.914Z'
    },
    {
      key: 's',
      value: 'CgAD4ACBfRS5+MjExNzQwN2ExNzQwYTc3MGM1MzViODRiZmZmOWE2OWEGcWWc',
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-24T15:30:04.916Z',
      lastAccessed: '2020-08-24T15:30:04.916Z'
    },
    {
      key: 'ebay',
      value: '%5Ejs%3D1%5Esbf%3D%23000000%5E',
      domain: 'ebay.com',
      path: '/',
      hostOnly: false,
      creation: '2020-08-24T15:30:04.916Z',
      lastAccessed: '2020-08-24T15:30:04.916Z'
    },
    {
      key: 'cid',
      value: 'XRa5T2Lj2LACEDp8%23857643373',
      expires: '2021-08-24T15:30:06.000Z',
      domain: 'ebay.com',
      path: '/',
      hostOnly: false,
      creation: '2020-08-24T15:30:04.917Z',
      lastAccessed: '2020-08-24T15:30:04.917Z'
    }
  ]
}

    That's all for now - sorry.

     

  • Michael S.

    Michael S. - 2020-08-25
     

  • Michael S.

    Michael S. - 2020-08-25

    Got the cookies.

    New code:

    const url = "https://signin.ebay.com/ws/eBayISAPI.dll?SignIn";
const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { window } = new JSDOM();
const resourceLoader = new jsdom.ResourceLoader({
  strictSSL: false,
  // MUST BE THE SAME LIKE ESNIPER AGENT !!!
  userAgent: "Mozilla/4.7 [en] (X11; U; Linux 2.2.12 i686)"
});

const options = {
  resources: resourceLoader,
  runScripts: 'dangerously',
  pretendToBeVisual: true,
  virtualConsole: new jsdom.VirtualConsole(),
  cookieJar: new jsdom.CookieJar(),
}

function cb(err, data) {
  console.log(data)
}

class NAVIGATOR {
   language = "en"
   cpuClass = "unknown"
   platform = "unknown"
   doNotTrack = "unknown"
   plugins = { }
   maxTouchPoints = 0
   appName = "unknown"
}

class SCREEN {
   width = 1024
   height = 768
   availHeight = 768
   availWidth = 1024
   pixelDepth = 16
}

const screen = new SCREEN();
const navigator = new NAVIGATOR();

JSDOM.fromURL(url,  options ).then(dom => 
{
    console.log("Loading " + url + " ok.")
});

setTimeout(() => {
   console.log("Dumping CookieJar:")
   options.cookieJar.serialize(cb)
}, 2000)

    Output:

    $ npm start

> ebay@1.0.0 start MYPATH/ebay
> node --trace-warnings signin.js


Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
Dumping CookieJar:
{
  version: 'tough-cookie@3.0.1',
  storeType: 'MemoryCookieStore',
  rejectPublicSuffixes: true,
  cookies: [
    {
      key: 'DG_IID',
      value: '86D467C1-4D04-3B8F-9881-3C28CFF8F237',
      maxAge: 2628000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.809Z',
      lastAccessed: '2020-08-25T14:55:31.809Z'
    },
    {
      key: 'DG_UID',
      value: '708D214B-3CEA-3330-A14B-5E743277BEB7',
      maxAge: 2628000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.815Z',
      lastAccessed: '2020-08-25T14:55:31.815Z'
    },
    {
      key: 'DG_ZID',
      value: 'AE5965D5-011C-3811-8414-B94C540368D7',
      maxAge: 2628000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.817Z',
      lastAccessed: '2020-08-25T14:55:31.817Z'
    },
    {
      key: 'DG_ZUID',
      value: '68B2F79B-0ABB-38C4-9220-4B304FF52B92',
      maxAge: 2628000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.819Z',
      lastAccessed: '2020-08-25T14:55:31.819Z'
    },
    {
      key: 'DG_HID',
      value: '77AF1B15-8905-3C4C-B6AE-4E9AE45C197F',
      maxAge: 2628000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.820Z',
      lastAccessed: '2020-08-25T14:55:31.820Z'
    },
    {
      key: 'DG_SID',
      value: 'MYIP:VycnAGq3lsNkaEhCQU1CLzOGwXZ53oiu/jfz7tZ+wC8',
      maxAge: 31536000,
      domain: 'ebay.com',
      path: '/',
      httpOnly: true,
      hostOnly: false,
      creation: '2020-08-25T14:55:31.822Z',
      lastAccessed: '2020-08-25T14:55:31.822Z'
    }
  ]
}

    Next steps:
    1. Write cookies to file system
    2. Try, if the cookies work with esniper

     

  • Michael S.

    Michael S. - 2020-08-27

    Next Update

    signin.js:

    const url = "https://signin.ebay.com/ws/eBayISAPI.dll?SignIn";
const jsdom = require("jsdom");
const { JSDOM } = jsdom;
const { window } = new JSDOM();
const resourceLoader = new jsdom.ResourceLoader({
  strictSSL: false,
  // MUST BE THE SAME LIKE ESNIPER AGENT !!!
  userAgent: "Mozilla/4.7 [en] (X11; U; Linux 2.2.12 i686)"
  //userAgent: "Mozilla/5.1 [en] (X11; U; Linux 2.2.12 i686)"
});

const options = {
  resources: resourceLoader,
  runScripts: 'dangerously',
  pretendToBeVisual: true,
  virtualConsole: new jsdom.VirtualConsole(),
  cookieJar: new jsdom.CookieJar(),
}

class NAVIGATOR {
   language = "en"
   cpuClass = "unknown"
   platform = "unknown"
   doNotTrack = "unknown"
   plugins = { }
   maxTouchPoints = 0
   appName = "unknown"
}

class SCREEN {
   width = 1024
   height = 768
   availHeight = 768
   availWidth = 1024
   pixelDepth = 16
}

const screen = new SCREEN();
const navigator = new NAVIGATOR();

JSDOM.fromURL(url, options).then(dom => 
{
    console.log("Loading " + url + " ok.")
});

function dump_cookieJar(data) {
   i=0
   console.log("Dumping CookieJar:")
   for(let c in data.cookies) {
    console.log(i++ + ":\tkey=" + data.cookies[c].key + "\tvalue=" + data.cookies[c].value)
   }
}

// Cookie collector
setInterval( () => {
   options.cookieJar.serialize( (err, data) => (function() { 
        s=0
        for(let c in data.cookies) {
          if( data.cookies[c].key == "DG_HID" )
            s = s | 0x0001
          else if( data.cookies[c].key == "DG_IID" )
            s = s | 0x0002
          else if( data.cookies[c].key == "DG_SID" )
            s = s | 0x0004
          else if( data.cookies[c].key == "DG_ZID" )
            s = s | 0x0008
          else if( data.cookies[c].key == "DG_UID" )
            s = s | 0x0010
          else if( data.cookies[c].key == "DG_ZUID" )
            s = s | 0x0020
          else if( data.cookies[c].key == "PS" )
            s = s | 0x0040
          else if( data.cookies[c].key == "ak_bmsc" )
            s = s | 0x0080
          else if( data.cookies[c].key == "c" )
            s = s | 0x0100
          else if( data.cookies[c].key == "cid" )
            s = s | 0x0200
          else if( data.cookies[c].key == "dp1" )
            s = s | 0x0400
          else if( data.cookies[c].key == "ebay" )
            s = s | 0x0800
          else if( data.cookies[c].key == "nonsession" )
            s = s | 0x1000
          else if( data.cookies[c].key == "npii" )
            s = s | 0x2000
          else if( data.cookies[c].key == "s" )
            s = s | 0x4000
          else if( data.cookies[c].key == "thx_guid" )
            s = s | 0x8000
                }
        // Done - all cookies received
        if( (s & 0xFFFF) == 0xFFFF ) {
            dump_cookieJar(data)
            process.exit()
        }
   })() )
}, 250)

// Timeout
setInterval( () => {
    console.log("Timed out. None of the required cookies received!")
    options.cookieJar.serialize( (err, data) => (function() { 
        dump_cookieJar(data)
        })() )
    process.exit()
}, 15000)

    Output:

    $ npm start

> ebay@1.0.0 start MYPATH/ebay
> node --trace-warnings signin.js

Loading https://signin.ebay.com/ws/eBayISAPI.dll?SignIn ok.
Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
Dumping CookieJar:
0:  key=dp1 value=bu1p/QEBfX0BAX19AQA**630a7178^pbf/%232000000000000000000000000000461293df8^tzo/1a45f481886^bl/DE630a7178^
1:  key=nonsession  value=BAQAAAXQoEGjGAAaAADMABWEpPfg1MzE3NwDKACBjCnF4MzE2OGRhNGQxNzQwYTc2NDc3Mzc5NmRkZmZmNDAyZTYAywABX0gRgDUS2DN3gYMtBqFERQXyd5VpZTMLFQ**
2:  key=s   value=CgAD4ACBfSVv2MzE2OGRhNGQxNzQwYTc2NDc3Mzc5NmRkZmZmNDAyZTa8J37V
3:  key=ebay    value=%5Ejs%3D1%5Esbf%3D%23%5E
4:  key=cid value=dW3bpU0j6a9GUQdh%231190046650
5:  key=npii    value=btguid/3168da4d1740a764773796ddfff402e6630a7179^cguid/3168de561740a6e559813477fd320a7c630a7179^
6:  key=c   value=e007c0f675e0f2723b76
7:  key=DG_IID  value=86D467C1-4D04-3B8F-9881-3C28CFF8F237
8:  key=DG_UID  value=2F5A932D-CF60-38B4-A98E-6FA1B1574BB6
9:  key=DG_ZID  value=AE5965D5-011C-3811-8414-B94C540368D7
10: key=DG_ZUID value=DC305FC4-588B-3EB3-B4D1-69D364BF18D3
11: key=DG_HID  value=2B4B26E3-D0B4-3598-AC02-5FB9D12C0366
12: key=DG_SID  value=IPADDR:j3HRwoS5nefrcEEktcVm5rbXmaqf300wB6adkhoi91k
13: key=ak_bmsc value=A27D58DF47DD86C8DF5C04925542BDA050434A1F093A0000770A485F84289846~ply6m9D7ymVz0HidmBW+4JLdKOUgeUwYRmHlI515wx1QziA+N3wy8amFWBs1qCMRgSBUBqSPAdZk3ypwWLptwGIad8qML5qC9+sT2mf9K+OmOUHEDRzrW/VOlg9xeyg+7X0NhjHj+/BXPw4fT4MZJpqx8+GAnckUt+2EmCs1eab7X33KjNVyvtJONNBpZPAL2X4ZXO9+KojvcnjIIx2uCpVhIhrhKhv8S1KmptAlbpYyM=
14: key=PS  value=T.0
15: key=thx_guid    value=e92dc6b1d857402eb3cfcdddf76c31ad

    Next steps:
    1. Write cookies to file system
    2. Try, if the cookies work with esniper, when read from a file**

     

  • Domenico

    Domenico - 2020-09-12

    Hello to everyone! :) I've been a regular (and really satisfied) esniper user for years and I don't know if it's going to help or not. I fear not (I'm sure it has been already considered as a viable option by Michael and probably it fails as Duktape does or it isn't just suitable) but I make a try anyway.
    What about cesanta/mjs? Still a c/c++ embedded js engine, it seems with a low footprint too.
    I don't know, I'm just trying to help. A nice day to everyone! :)

     

    Last edit: Domenico 2020-09-12

  • Michael S.

    Michael S. - 2020-09-26

    I think, cesanta/mjs isnt suitable to solve our problem. The engine must be able to do XMLHttpRequest requestes (see https://signin.ebay.com/nkfytkqtoxtljvzb.js). It also must have full access to the DOM, which currently is a problem with nodejs.

     

  • Rob Frohne

    Rob Frohne - 2020-10-29

    Thanks Michael,
    Does this look like the end for esniper?
    Rob

     

  • Michael S.

    Michael S. - 2020-10-29

    Currently I'm to busy to investigate this issue.
    Integration a JS engine would definitely the end of a "lightweight ebay sniper".

     

  • Ulrich Teichert

    Ulrich Teichert - 2021-01-02

    Hi, don't know if this would fit your requirements: https://github.com/yodaos-project/ShadowNode

     

Log in to post a comment.