Xowa-Webserver: Issues, requests and progress thread

mattze96
2013-12-04
2014-05-01
  • mattze96

    mattze96 - 2013-12-04

    Hi,

    in this thread you can post everything according the Webserver for xowa, which I'm developing together with gnosygnu. It is working now and you can read some wikis fine.

    I will also do some short status videos, every time I add something new.

    Now the webserver is not included in the xowa release, because it is still unstable.
    If you know what you are doing you can extract the files from this repo to your dev-folder. At this moment there a three files which will be overwritten.

    https://github.com/Mattze96/XowaWebserver

    If you have any questions or suggestions you can post them below. (If you're not good at english you can write german too ;-) )

    Matthias


    My blog: https://mattze96.safe-ws.de/blog/
    My Youtube Channel: https://www.youtube.com/user/Mattze96
    Initial discussion thread: https://sourceforge.net/p/xowa/discussion/general/thread/c54b9853/

     
    • mattze96

      mattze96 - 2013-12-09
      • Help:Diagnostics/Javascript/All should display everything fine now.
        (I fixed a path error for the MathJax.js, because it had a parameter in the filename. /path/MathJax.js?=default. This fix is working for other files too.)

      [The newest development version is alway available on my github repository]

       
      • gnosygnu

        gnosygnu - 2013-12-10

        Hey! That works very well. Nice going!

         
  • gnosygnu

    gnosygnu - 2013-12-05

    Hey, thanks for putting this up.

    Depending on how my weekend goes, I may try to integrate this into v0.12.1. I'll provide a separate command-line argument for it, so that it can co-exist with the Firefox Add-On.

     
  • mattze96

    mattze96 - 2013-12-11

    Hi,

    should i add a feature, so that you can specify a "custom" main-page for you webserver. So if you request only http://localhost you get forwarded to for example to http://localhost/welcome and not to the main-page of the xowa-wiki.

    Matthias

     
    • gnosygnu

      gnosygnu - 2013-12-12

      Sure, but I think that would have to be specified "server-side"? For example, I'm thinking starting the xowa server with "java -jar xowa_linux.jar --app_mode http_server --http_server_redirect_main welcome"

       
      • Anonymous - 2013-12-12

        I tought also the ability to create a own html file and refer to it :)

         
        • gnosygnu

          gnosygnu - 2013-12-13

          Hmmm... I'd prefer to have the server go to XOWA home or Wikipedia pages.

          That said, attach a mock-up and we'll see what's involved.

          Thanks.

           
  • gnosygnu

    gnosygnu - 2013-12-13

    Incidentally, the http_server does not work on windows 64 or linux 64: http://www.reddit.com/r/xowa/comments/1sftan/release_minor_v0121_offline_image_databases_for/cdzun4u

    I was able to reproduce this on my windows 7 64-bit machine. Note that it works on my windows xp 32-bit machine.

    mattze96: The issue seems to be threading and the 64-bit OS. I was able to work around the issue by doing the following:

    // bad
    String cmd = app.Url_converter_url().Decode_str(url_encoded_str);
    
    // good
    Url_encoder url_converter = Url_encoder.new_http_url_();
    String cmd = url_converter.Decode_str(url_encoded_str);
    

    The app.Url_converter() is storing state, that causes the threaded calls to misbehave.

    I'll track this down for this upcoming release. Note that I'm in the process of changing a lot of other local state behavior b/c of Scribunto recursive calls.

     
  • tibo1902

    tibo1902 - 2014-02-28

    Hi,

    Thanks for this great feature. But i have problem to make it works.

    I can start xowa (v 1.2.4.1) in server mode with :

    java -Xmx256m -jar $XOWA_DIR/xowa_linux_64.jar --root_dir $XOWA_DIR/ --app_mode server 
    

    It starts :

    server started: listening on 55000. Press Ctrl+C to exit

    But when i tried to access with my browser (chromium), th server return this error :

    processing cmd: version must be 0; version:71
    server error: 0 unsupported encoding
    
    1   class java.lang.NullPointerException null <java.lang.NullPointerException>
    

    I'm on linux 64 bits.

     
  • gnosygnu

    gnosygnu - 2014-02-28

    Hi. I think you posted this twice, but just to repeat my answer from the other thread

    Your command-line args are for the firefox addon. They require that you use firefox, and a url of "xowa://home" or "xowa://simple.wikipedia.org/wiki/Main_Page". See home/wiki/Help:XOWA_viewer

    I think you want the http server. You should look at home/wiki/Help:Http_Server in XOWA. To summarize, you'd have to do the following:

    • Launch XOWA with java -jar /xowa/xowa_linux.jar --app_mode http_server --http_server_port 8080
    • Navigate your browser to localhost:8080

    Both the Firefox Addon and the HTTP Server have been tested on Ubuntu, so hopefully you should be okay.

     
  • Anonymous - 2014-04-28

    Hi. I've just finished installing Xowa en.wikipedia.org in Windows 2000 offline, with all 89 fsdb.main sql's. The Windows build took a VERY long time (over 100 hours), but it works well: The only thing missing is the figures.
    I was able to open a page with a figure using wikipedia online, copy the figure to my offline installation, and install it where the offline (missing) figure was looking for it: Again, worked fine!
    HOWEVER, when I tried the same thing with a bigger file (Oregon state posting), I could not locate the copied figure so it would display.

    Can someone help me: Where should I put the figure in my offline XowA installation?
    Details: Offline XowA says it is finding the downloaded, figureless display at "en.wikipedia.org/wiki/Oregon". I am trying to display the upper-right "Flag of Oregon" which according to XowA, by clicking on it, is "could not find: file:///wiki/File:Flag_of_oregon.svg". Without clicking, XowA says "File:Flag of Oregon.svg" (no underscores). The .svg file is OK, because it displays correctly, using Corel Draw.

     
  • gnosygnu

    gnosygnu - 2014-04-28

    Hi. Thanks for the details. I'm going to respond piece-meal, but let me know if I missed anything.

    The Windows build took a VERY long time (over 100 hours)

    I assume you mean downloading the 89 fsdb.main files took over 100 hours? Out of curiosity, did you download by direct download or by torrent? And are you in the United States? If elsewhere, where approximately?

    but it works well:

    Cool. Thanks for confirming.

    The only thing missing is the figures.

    Sorry to hear that. Let's see if we can get to the bottom of it.

    I am trying to display the upper-right "Flag of Oregon" which according to XowA, by clicking on it, is "could not find: file:///wiki/File:Flag_of_oregon.svg".

    This is a slightly different issue. There's an entry for this in the FAQ (home/wiki/Help:FAQs), but I'll change this message in the next XOWA build.

    In short, you need to install commons.wikimedia.org in order to click on an image. A longer explanation is at home/wiki/Help:Wikis/Commons

    Where should I put the figure in my offline XowA installation?

    The figures (images) should automatically extract if you have the fsdb.main files set up.

    Let's assume you've XOWA extracted to C:\xowa where your xowa.exe is C:\xowa\xowa.exe

    • Your wiki databases for en.wikipedia.org should be at C:\xowa\wiki\en.wikipedia.org\ where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org\en.wikipedia.org.000.sqlite3 and several others (.001, .002, etc..)
    • Your file databases (for the figures / images) should be at C:\xowa\file\en.wikipedia.org where you'll have the following layout
      • C:\xowa\file\en.wikipedia.org\wiki.mnt.sqlite3
      • C:\xowa\file\en.wikipedia.org\wiki.orig#00.sqlite3
      • C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.abc.sqlite3
      • C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.atr.00.sqlite3
      • C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.bin.0000.sqlite3 -> C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.bin.0089.sqlite3

    So, if these files are not in these locations, then you'll run into issues.

    If they are at these locations, then something else is going on.

    • Is File retrieval checked? See home/wiki/Help:Options/Files
    • Do you have enough space / read-write access to your hard-drive? (99% likely that you do, but just to make sure)
    • It may be a Windows 2000 issue. You can check your log file at C:\xowa\user\anonymous\app\tmp\log, or attach relevant portions here

    Lastly, as a matter of explanation, this is what XOWA does if everything is set up correctly:

    • Looks up the image in wiki.orig#00.sqlite3 (needed for general image metadata as well as redirects)
    • Looks up the image in fsdb.atr.00.sqlite3 (needed for locating the "bin" db)
    • Extracts the image and copies it to C:/xowa/file/
    • Generates HTML like the following:
    &lt;td style="width: 130px; vertical-align: middle;">&lt;a href="/wiki/File:Flag_of_Oregon.svg" class="image" title="Flag of Oregon" xowa_title="Flag of Oregon.svg">&lt;img id="xowa_file_img_-1" alt="Flag of Oregon" src="file:///C:/xowa/file/commons.wikimedia.org/thumb/b/9/6/1/Flag_of_Oregon.svg/125px.png" width="125" height="75" class="thumbborder" />&lt;/a>
    

    Hope this helps.

    Thanks.

     
  • Anonymous - 2014-04-29
    The Windows build took a VERY long time (over 100 hours)
    

    I assume you mean downloading the 89 fsdb.main files took over 100 hours?

    ANS: No; the runtime of xowa_build_windows.bat took over 100 hours on win2k (2 GHz machine). As I recall, the "Assign pages to categories" was slowest, but I may be mistaken.

    Your wiki databases for en.wikipedia.org should be at C:\xowa\wiki\en.wikipedia.org\ [ANS: Yes] where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org [ANS: No, no such "J:gplx..." file] \en.wikipedia.org.000.sqlite3 and several others (.001, .002, etc. [ANS: Yes, .000.sqlite3 thru 008*]

    Out of curiosity, did you download by direct download or by torrent? And are you in the United States?

    ANS: Direct. I only use Linus online, so torrent didn't seem to be available. I am in Oregon, USA.

    Your file databases (for the figures / images) should be at C:\xowa\file\en.wikipedia.org where you'll have the following layout
    C:\xowa\file\en.wikipedia.org\wiki.mnt.sqlite3
    C:\xowa\file\en.wikipedia.org\wiki.orig#00.sqlite3
    ANS: I do not have these two, but I have all the others you listed. HOW CAN I CORRECT THIS?
    ...
    If they are at these locations, then something else is going on [ANS: They are; the "Oregon" errors all seem to be because XowA is trying to access wikipedia online, and my Windows machine has no online connection.

     
  • gnosygnu

    gnosygnu - 2014-04-30

    ANS: No; the runtime of xowa_build_windows.bat took over 100 hours on win2k (2 GHz machine).

    Wow. That's abysmally slow. Kudos to you for letting it complete.

    I don't know why it's that bad. My best guess is that it's SQLite and Windows 2000. I found this link, but it's rather inconclusive: http://sqlite.1065341.n5.nabble.com/Huge-performance-difference-between-SQLite-on-Windows-XP-and-Windows-2000-td11357.html

    Out of curiosity, what are your hard drive specs? If you have a very slow hard drive (4200 RPM) or little free space / much defragmenting, then this would definitely slow it down. You would see this poor performance everywhere though.

    [ANS: Yes] where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org [ANS: No, no such "J:gplx..." file]

    Sorry. That was a typo. It should be C:\xowa\wiki\en.wikipedia.org just like the others.

    ANS: Direct. I only use Linus online, so torrent didn't seem to be available. I am in Oregon, USA.

    Ok. Thanks. I'll resume posting torrents for future en.wikipedia.org dumps then. I don't know if torrenting will be much faster, but at least you're continental US (I'm East Coast). Hopefully, it should be better than 100 hours.

    ANS: I do not have these two, but I have all the others you listed. HOW CAN I CORRECT THIS?

    I'm guessing you didn't download this: https://archive.org/details/Xowa_enwiki_2013-11-04_images_complete_00. This will have wiki.mnt.sqlite3, wiki.orig#00.sqlite3. I downloaded it just now, and confirmed that they are there.

    Also, you may want to look at the latest set: https://archive.org/details/Xowa_enwiki_2014-04-02_images_update. This will have files 090 and 091 (for the April updates)

    [ANS: They are; the "Oregon" errors all seem to be because XowA is trying to access wikipedia online, and my Windows machine has no online connection.

    Okay. That helps. Let's see if the above works.

     
  • Anonymous - 2014-05-01

    OK! That worked, and I now can see all the images when I view a wikipedia topic!
    Thanks very much for your help!
    One remaining question: I copied and installed a directory named ".../file/en.wikipedia.org/fsdb.update_00".
    Should that be renamed to ".../fsdb.update" (without the "_00")?

     
  • gnosygnu

    gnosygnu - 2014-05-01

    OK! That worked, and I now can see all the images when I view a wikipedia topic!

    Cool. Thanks for confirming, and glad to hear that it's working fine.

    Should that be renamed to ".../fsdb.update" (without the "_00")?

    Nope. It should be "fsdb.update_00" (with the "_00").

    Let me know if there is anything else.

    Thanks.

     


Anonymous

Cancel  Add attachments