xowa / Discussion / General Discussion: Xowa-Webserver: Issues, requests and progress thread

mattze96 - 2013-12-04

Hi,

in this thread you can post everything according the Webserver for xowa, which I'm developing together with gnosygnu. It is working now and you can read some wikis fine.

I will also do some short status videos, every time I add something new.

Now the webserver is not included in the xowa release, because it is still unstable.
If you know what you are doing you can extract the files from this repo to your dev-folder. At this moment there a three files which will be overwritten.

https://github.com/Mattze96/XowaWebserver

If you have any questions or suggestions you can post them below. (If you're not good at english you can write german too ;-) )

Matthias

My blog: https://mattze96.safe-ws.de/blog/
My Youtube Channel: https://www.youtube.com/user/Mattze96
Initial discussion thread: https://sourceforge.net/p/xowa/discussion/general/thread/c54b9853/

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.
- mattze96 - 2013-12-09
  
  Help:Diagnostics/Javascript/All should display everything fine now.
  (I fixed a path error for the MathJax.js, because it had a parameter in the filename. /path/MathJax.js?=default. This fix is working for other files too.)
  
  [The newest development version is alway available on my github repository]
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Anonymous
    
    Add attachments
    Cancel
    You seem to have CSS turned off. Please don't fill out this field.
    
    You seem to have CSS turned off. Please don't fill out this field.
  - gnosygnu - 2013-12-10
    
    Hey! That works very well. Nice going!
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
    
    Anonymous
    
    Add attachments
    Cancel
    You seem to have CSS turned off. Please don't fill out this field.
    
    You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2013-12-05

Hey, thanks for putting this up.

Depending on how my weekend goes, I may try to integrate this into v0.12.1. I'll provide a separate command-line argument for it, so that it can co-exist with the Firefox Add-On.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

mattze96 - 2013-12-11

Hi,

should i add a feature, so that you can specify a "custom" main-page for you webserver. So if you request only http://localhost you get forwarded to for example to http://localhost/welcome and not to the main-page of the xowa-wiki.

Matthias

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.
- gnosygnu - 2013-12-12
  
  Sure, but I think that would have to be specified "server-side"? For example, I'm thinking starting the xowa server with "java -jar xowa_linux.jar --app_mode http_server --http_server_redirect_main welcome"
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Anonymous
    
    Add attachments
    Cancel
    You seem to have CSS turned off. Please don't fill out this field.
    
    You seem to have CSS turned off. Please don't fill out this field.
  - Anonymous - 2013-12-12
    
    I tought also the ability to create a own html file and refer to it :)
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
    
    Anonymous
    
    Add attachments
    Cancel
    You seem to have CSS turned off. Please don't fill out this field.
    
    You seem to have CSS turned off. Please don't fill out this field.
    - gnosygnu - 2013-12-13
      
      Hmmm... I'd prefer to have the server go to XOWA home or Wikipedia pages.
      
      That said, attach a mock-up and we'll see what's involved.
      
      Thanks.
      
      If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
      
      Anonymous
      
      Add attachments
      Cancel
      You seem to have CSS turned off. Please don't fill out this field.
      
      You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2013-12-13

Incidentally, the http_server does not work on windows 64 or linux 64: http://www.reddit.com/r/xowa/comments/1sftan/release_minor_v0121_offline_image_databases_for/cdzun4u

I was able to reproduce this on my windows 7 64-bit machine. Note that it works on my windows xp 32-bit machine.

mattze96: The issue seems to be threading and the 64-bit OS. I was able to work around the issue by doing the following:

:::java // bad String cmd = app.Url_converter_url().Decode_str(url_encoded_str); // good Url_encoder url_converter = Url_encoder.new_http_url_(); String cmd = url_converter.Decode_str(url_encoded_str);

The app.Url_converter() is storing state, that causes the threaded calls to misbehave.

I'll track this down for this upcoming release. Note that I'm in the process of changing a lot of other local state behavior b/c of Scribunto recursive calls.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

tibo1902 - 2014-02-28

Hi,

Thanks for this great feature. But i have problem to make it works.

I can start xowa (v 1.2.4.1) in server mode with :

java -Xmx256m -jar $XOWA_DIR/xowa_linux_64.jar --root_dir $XOWA_DIR/ --app_mode server

It starts :

server started: listening on 55000. Press Ctrl+C to exit

But when i tried to access with my browser (chromium), th server return this error :

processing cmd: version must be 0; version:71 server error: 0 unsupported encoding 1 class java.lang.NullPointerException null <java.lang.NullPointerException>

I'm on linux 64 bits.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2014-02-28

Hi. I think you posted this twice, but just to repeat my answer from the other thread

Your command-line args are for the firefox addon. They require that you use firefox, and a url of "xowa://home" or "xowa://simple.wikipedia.org/wiki/Main_Page". See home/wiki/Help:XOWA_viewer

I think you want the http server. You should look at home/wiki/Help:Http_Server in XOWA. To summarize, you'd have to do the following:

Launch XOWA with java -jar /xowa/xowa_linux.jar --app_mode http_server --http_server_port 8080

Navigate your browser to localhost:8080

Both the Firefox Addon and the HTTP Server have been tested on Ubuntu, so hopefully you should be okay.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

Anonymous - 2014-04-28

Hi. I've just finished installing Xowa en.wikipedia.org in Windows 2000 offline, with all 89 fsdb.main sql's. The Windows build took a VERY long time (over 100 hours), but it works well: The only thing missing is the figures.
I was able to open a page with a figure using wikipedia online, copy the figure to my offline installation, and install it where the offline (missing) figure was looking for it: Again, worked fine!
HOWEVER, when I tried the same thing with a bigger file (Oregon state posting), I could not locate the copied figure so it would display.

Can someone help me: Where should I put the figure in my offline XowA installation?
Details: Offline XowA says it is finding the downloaded, figureless display at "en.wikipedia.org/wiki/Oregon". I am trying to display the upper-right "Flag of Oregon" which according to XowA, by clicking on it, is "could not find: file:///wiki/File:Flag_of_oregon.svg". Without clicking, XowA says "File:Flag of Oregon.svg" (no underscores). The .svg file is OK, because it displays correctly, using Corel Draw.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2014-04-28

Hi. Thanks for the details. I'm going to respond piece-meal, but let me know if I missed anything.

The Windows build took a VERY long time (over 100 hours)

I assume you mean downloading the 89 fsdb.main files took over 100 hours? Out of curiosity, did you download by direct download or by torrent? And are you in the United States? If elsewhere, where approximately?

but it works well:

Cool. Thanks for confirming.

The only thing missing is the figures.

Sorry to hear that. Let's see if we can get to the bottom of it.

I am trying to display the upper-right "Flag of Oregon" which according to XowA, by clicking on it, is "could not find: file:///wiki/File:Flag_of_oregon.svg".

This is a slightly different issue. There's an entry for this in the FAQ (home/wiki/Help:FAQs), but I'll change this message in the next XOWA build.

In short, you need to install commons.wikimedia.org in order to click on an image. A longer explanation is at home/wiki/Help:Wikis/Commons

Where should I put the figure in my offline XowA installation?

The figures (images) should automatically extract if you have the fsdb.main files set up.

Let's assume you've XOWA extracted to C:\xowa where your xowa.exe is C:\xowa\xowa.exe

Your wiki databases for en.wikipedia.org should be at C:\xowa\wiki\en.wikipedia.org\ where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org\en.wikipedia.org.000.sqlite3 and several others (.001, .002, etc..)

Your file databases (for the figures / images) should be at C:\xowa\file\en.wikipedia.org where you'll have the following layout

C:\xowa\file\en.wikipedia.org\wiki.mnt.sqlite3

C:\xowa\file\en.wikipedia.org\wiki.orig#00.sqlite3

C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.abc.sqlite3

C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.atr.00.sqlite3

C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.bin.0000.sqlite3 -> C:\xowa\file\en.wikipedia.org\fsdb.main\fsdb.bin.0089.sqlite3

So, if these files are not in these locations, then you'll run into issues.

If they are at these locations, then something else is going on.

Is File retrieval checked? See home/wiki/Help:Options/Files

Do you have enough space / read-write access to your hard-drive? (99% likely that you do, but just to make sure)

It may be a Windows 2000 issue. You can check your log file at C:\xowa\user\anonymous\app\tmp\log, or attach relevant portions here

Lastly, as a matter of explanation, this is what XOWA does if everything is set up correctly:

Looks up the image in wiki.orig#00.sqlite3 (needed for general image metadata as well as redirects)

Looks up the image in fsdb.atr.00.sqlite3 (needed for locating the "bin" db)

Extracts the image and copies it to C:/xowa/file/

Generates HTML like the following:

<td style="width: 130px; vertical-align: middle;"><a href="/wiki/File:Flag_of_Oregon.svg" class="image" title="Flag of Oregon" xowa_title="Flag of Oregon.svg"><img id="xowa_file_img_-1" alt="Flag of Oregon" src="file:///C:/xowa/file/commons.wikimedia.org/thumb/b/9/6/1/Flag_of_Oregon.svg/125px.png" width="125" height="75" class="thumbborder" /></a>

Hope this helps.

Thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

Anonymous - 2014-04-29

The Windows build took a VERY long time (over 100 hours)

I assume you mean downloading the 89 fsdb.main files took over 100 hours?

ANS: No; the runtime of xowa_build_windows.bat took over 100 hours on win2k (2 GHz machine). As I recall, the "Assign pages to categories" was slowest, but I may be mistaken.

Your wiki databases for en.wikipedia.org should be at C:\xowa\wiki\en.wikipedia.org\ [ANS: Yes] where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org [ANS: No, no such "J:gplx..." file] \en.wikipedia.org.000.sqlite3 and several others (.001, .002, etc. [ANS: Yes, .000.sqlite3 thru 008*]

Out of curiosity, did you download by direct download or by torrent? And are you in the United States?

ANS: Direct. I only use Linus online, so torrent didn't seem to be available. I am in Oregon, USA.

Your file databases (for the figures / images) should be at C:\xowa\file\en.wikipedia.org where you'll have the following layout
C:\xowa\file\en.wikipedia.org\wiki.mnt.sqlite3
C:\xowa\file\en.wikipedia.org\wiki.orig#00.sqlite3
ANS: I do not have these two, but I have all the others you listed. HOW CAN I CORRECT THIS?
...
If they are at these locations, then something else is going on [ANS: They are; the "Oregon" errors all seem to be because XowA is trying to access wikipedia online, and my Windows machine has no online connection.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2014-04-30

ANS: No; the runtime of xowa_build_windows.bat took over 100 hours on win2k (2 GHz machine).

Wow. That's abysmally slow. Kudos to you for letting it complete.

I don't know why it's that bad. My best guess is that it's SQLite and Windows 2000. I found this link, but it's rather inconclusive: http://sqlite.1065341.n5.nabble.com/Huge-performance-difference-between-SQLite-on-Windows-XP-and-Windows-2000-td11357.html

Out of curiosity, what are your hard drive specs? If you have a very slow hard drive (4200 RPM) or little free space / much defragmenting, then this would definitely slow it down. You would see this poor performance everywhere though.

[ANS: Yes] where you'll have a file called J:\gplx\xowa\wiki\en.wikipedia.org [ANS: No, no such "J:gplx..." file]

Sorry. That was a typo. It should be C:\xowa\wiki\en.wikipedia.org just like the others.

ANS: Direct. I only use Linus online, so torrent didn't seem to be available. I am in Oregon, USA.

Ok. Thanks. I'll resume posting torrents for future en.wikipedia.org dumps then. I don't know if torrenting will be much faster, but at least you're continental US (I'm East Coast). Hopefully, it should be better than 100 hours.

ANS: I do not have these two, but I have all the others you listed. HOW CAN I CORRECT THIS?

I'm guessing you didn't download this: https://archive.org/details/Xowa_enwiki_2013-11-04_images_complete_00. This will have wiki.mnt.sqlite3, wiki.orig#00.sqlite3. I downloaded it just now, and confirmed that they are there.

Also, you may want to look at the latest set: https://archive.org/details/Xowa_enwiki_2014-04-02_images_update. This will have files 090 and 091 (for the April updates)

[ANS: They are; the "Oregon" errors all seem to be because XowA is trying to access wikipedia online, and my Windows machine has no online connection.

Okay. That helps. Let's see if the above works.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

Anonymous - 2014-04-30

You wrote:
...
I'm guessing you didn't download this: https://archive.org/details/Xowa_enwiki_2013-11-04_images_complete_00. This will have wiki.mnt.sqlite3, wiki.orig#00.sqlite3. I downloaded it just now, and confirmed that they are there.

ANS: I can't seem to download the above: I get "File not available". Are you sure your path "...complete_00." is correct?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.
- gnosygnu - 2014-04-30
  
  Sorry. Remove the period: https://archive.org/details/Xowa_enwiki_2013-11-04_images_complete_00
  
  Also, the same for this link: https://archive.org/details/Xowa_enwiki_2014-04-02_images_update
  
  Finally, this link has all known locations:
  http://xowa.sourceforge.net/image_dbs.html
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Anonymous
    
    Add attachments
    Cancel
    You seem to have CSS turned off. Please don't fill out this field.
    
    You seem to have CSS turned off. Please don't fill out this field.

Anonymous - 2014-05-01

OK! That worked, and I now can see all the images when I view a wikipedia topic!
Thanks very much for your help!
One remaining question: I copied and installed a directory named ".../file/en.wikipedia.org/fsdb.update_00".
Should that be renamed to ".../fsdb.update" (without the "_00")?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

gnosygnu - 2014-05-01

OK! That worked, and I now can see all the images when I view a wikipedia topic!

Cool. Thanks for confirming, and glad to hear that it's working fine.

Should that be renamed to ".../fsdb.update" (without the "_00")?

Nope. It should be "fsdb.update_00" (with the "_00").

Let me know if there is anything else.

Thanks.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Anonymous
  
  Add attachments
  Cancel
  You seem to have CSS turned off. Please don't fill out this field.
  
  You seem to have CSS turned off. Please don't fill out this field.

Xowa-Webserver: Issues, requests and progress thread

A free, open-source, offline Wikipedia application

Forums

Help

Xowa-Webserver: Issues, requests and progress thread

Xowa-Webserver: Issues, requests and progress thread

A free, open-source, offline Wikipedia application

Forums

Help

Xowa-Webserver: Issues, requests and progress thread document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Xowa-Webserver: Issues, requests and progress thread