#21 Japanese auction sites

open
nobody
None
5
2007-02-13
2007-02-13
Rolf
No

How about a version for Japanese auction sites? I could provide the needed input (and would be a user myself).

Discussion

  • Scott Nicol
    Scott Nicol
    2007-02-13

    Logged In: YES
    user_id=441678
    Originator: NO

    If you want to retarget go ahead. I could help, but I don't have the time to take the lead. And as for retargeting, it's been done before. I wrote esniper in 1996 to snipe auctions on OnSale.

    You'll need to rewrite the parsing, and maybe the overall strategy (depending on the rules of the auction site).

    The code is undergoing some changes now which will hopefully simplify changing the parsing.

     
  • Scott Nicol
    Scott Nicol
    2007-02-13

    Logged In: YES
    user_id=441678
    Originator: NO

    Moving to feature request, as it isn't a bug...

     
  • Rolf
    Rolf
    2008-03-16

    Logged In: YES
    user_id=1159152
    Originator: YES

    Scott, what do you need to know as far as parsing and auction strategy is concerned?

     
  • Scott Nicol
    Scott Nicol
    2008-03-17

    Logged In: YES
    user_id=441678
    Originator: NO

    If you can point me to the auction site (and assuming I can navigate it), I can give advice. Implementation would have to be done by somebody else.

     
  • Rolf
    Rolf
    2008-03-17

    Logged In: YES
    user_id=1159152
    Originator: YES

    Thank you for the quick reply.

    I guess the two most important sites would be

    http://auction.yahoo.co.jp/
    http://auction.rakuten.co.jp/

     
  • Scott Nicol
    Scott Nicol
    2008-03-17

    Logged In: YES
    user_id=441678
    Originator: NO

    Both look possible to do.

    There are five main things you need to do:

    1. Get auction details. On eBay I use the bid history page, as that seems the most stable. Both of these sites appear to have the item information (price, end time, high bid, etc) in clearly set out tables, which (hopefully) isn't too hard to find. Just search for the text on the first row, or something like that.

    2. Login. I didn't look at this on these sites, because I don't have a login.

    3. Bid. On ebay it's a two-step process, which I call pre-bid and bid. You have to pick up some information off the pre-bid result and pass it into bid. I have no idea how these sites do it, if it's 1 step it would be easier.

    4. Handle results. Save a bunch of pages. Item with no bids. Item with bids. Item where you bid. Item ended. Invalid item. Bid too low. Bid too late. Bid successful. Login failed. Login successful. And whatever else you can think of. Pass these saved pages through your parser. On esniper I use the hidden -X option, which reads a file from stdin and processes it like it is a page retrieved from the site. The number of times -X is specified tells esniper what kind of parsing to do. For example, esniper -XX reads a bid history from stdin.

    5. Multi-byte and/or wide character. A long time ago I worked on writing unix for Fujitsu. Although many Japanese files (and I assume web pages) are in a multi-byte encoding, it's not so easy to search multi-byte. So as I recall you read with whatever encoding was used, then convert to wide character for use internally, then write with whatever output encoding is chosen. The problem is that esniper is ASCII, so it will need a lot of work to convert it to work with multibyte and wide characters.

    You may be better off using the ideas from esniper and writing a new, clean implementation. If I was writing esniper from scratch (instead of maintaining a program I wrote in 1995), I wouldn't write it in C. I'd probably use Java, as that's what I'm most familiar with. But anything with good wide/multibyte support and an HTML library would do.

     
  • Rolf
    Rolf
    2008-03-17

    Logged In: YES
    user_id=1159152
    Originator: YES

    Thank you, Scott, for the todo list. It kind of reads like you suggest to write a new tool. If anything, I'd be only capable of tweaking esniper. And even for that, I'd need your help to some extent (I cannot code in C).

    I'll start working on 1-4, others are of course welcome to join. Not sure if 5 is any problem. For example, esniper currently does not handle Umlauts, either. But that is only a cosmetic problem, the core function does not need anything out of ASCII.

    1. What information does esniper need to extract? You already mentioned price, end time and current high bid. Does esniper understand regexp?

    2. I have not logged in either, yet. I'll get myself an account.

    rest to be done later.

     
  • Scott Nicol
    Scott Nicol
    2008-03-17

    Logged In: YES
    user_id=441678
    Originator: NO

    First off, it's a lot of work I know I won't have time. I still have esniper feature requests (some of which I wrote!) that are many years old. If you can find somebody who has the time to write it, you can refer them to me for help.

    multibyte and/or wide character is a problem. ASCII works fine with German because most characters fit within 7-bit ASCII, with only a few in the 8th bit.

    Kanji has thousands of characters, so most encodings for Japanese are 16 bit. Some encodings are multibyte, which is a sequence of 8 bit bytes where each character may be composed of 1, 2 or 3 bytes. OK for input and output, but hell to manipulate inside a program.

    Java has good support for Japanese, and so does C#. I'm sure most other newer languages do, too.

    There may be sniping programs already written for these sites.

    Also, it appears that Yahoo auctions has auto extension:

    http://nichidoo.com/

    An auction can't end until there are no bids for 5 minutes. That changes sniping strategy. You program the sniper to wait until a few seconds are left, then bid 1 increment higher. Then again wait until a few seconds are left, if you aren't high bidder bid again. Repeat until:

    1. You win
    2. You hit your max bid
    3. Human opponent falls asleep

    This strategy worked well with onsale, which is the auction site I originally wrote esniper to target (ebay didn't exist then).