Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Problem with compiling zimwriterfs - missing magic.h

Help
2014-02-06
2014-02-27
  • franc walter
    franc walter
    2014-02-06

    Hello
    I just try to compile zimwriterfs.cpp on my Ubuntu 12.04 server.
    First I compiled liblzma and libzim without problems.
    I didn't compile libmagic because I have it already on my system (libmagic1 - 5.09-2) and I didn't find the sources on the url in the README file of zimwriterfs (ftp://ftp.astron.com/pub/file/ is not reachable).
    Now when I do '/.configure' no errors. But when I do 'make', I get

    # make
    g++ -DPACKAGE_NAME=\"zimwriterfs\" -DPACKAGE_TARNAME=\"zimwriterfs\" -DPACKAGE_VERSION=\"0.1\" -DPACKAGE_STRING=\"zimwriterfs\ 0.1\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"zimwriterfs\" -DVERSION=\"0.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1     -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STAT64=1 -DCLUSTER_CACHE_SIZE=16 -DDIRENT_CACHE_SIZE=512 -DLZMA_MEMORY_SIZE=128 -DENABLE_LZMA=1 -I.      -O3 -Igumbo -g -O2 -MT zimwriterfs-zimwriterfs.o -MD -MP -MF      .deps/zimwriterfs-zimwriterfs.Tpo -c -o     zimwriterfs-zimwriterfs.o `test -f 'zimwriterfs.cpp' || echo './'`zimwriterfs.cpp
    zimwriterfs.cpp:21:19: schwerwiegender Fehler: magic.h: Datei oder Verzeichnis nicht gefunden
    Kompilierung beendet.
    make: *** [zimwriterfs-zimwriterfs.o] Fehler 1
    

    .
    I have the magic.h in:

    /usr/include/linux/magic.h
    

    so I tried to put this whole path into the zimwriterfs.cpp:

    #include </usr/include/linux/magic.h>
    

    but then I even get worse errors:

    # make
    g++ -DPACKAGE_NAME=\"zimwriterfs\" -DPACKAGE_TARNAME=\"zimwriterfs\" -DPACKAGE_VERSION=\"0.1\" -DPACKAGE_STRING=\"zimwriterfs\ 0.1\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"zimwriterfs\" -DVERSION=\"0.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STAT64=1 -DCLUSTER_CACHE_SIZE=16 -DDIRENT_CACHE_SIZE=512 -DLZMA_MEMORY_SIZE=128 -DENABLE_LZMA=1 -I.      -O3 -Igumbo -g -O2 -MT zimwriterfs-zimwriterfs.o -MD -MP -MF .deps/zimwriterfs-zimwriterfs.Tpo -c -o zimwriterfs-zimwriterfs.o `test -f 'zimwriterfs.cpp' || echo './'`zimwriterfs.cpp
    zimwriterfs.cpp:53:1: Fehler: »magic_t« bezeichnet keinen Typ
    zimwriterfs.cpp: In Funktion »std::string getMimeTypeForFile(const string&)«:
    zimwriterfs.cpp:338:39: Fehler: »magic« wurde in diesem Gültigkeitsbereich nicht definiert
    zimwriterfs.cpp:338:58: Fehler: »magic_file« wurde in diesem Gültigkeitsbereich nicht definiert
    zimwriterfs.cpp: In Funktion »int main(int, char**)«:
    zimwriterfs.cpp:666:3: Fehler: »magic« wurde in diesem Gültigkeitsbereich nicht definiert
    zimwriterfs.cpp:666:22: Fehler: »MAGIC_MIME« wurde in diesem Gültigkeitsbereich nicht definiert
    zimwriterfs.cpp:666:32: Fehler: »magic_open« wurde in diesem Gültigkeitsbereich nicht definiert
    zimwriterfs.cpp:667:25: Fehler: »magic_load« wurde in diesem Gültigkeitsbereich nicht definiert
    make: *** [zimwriterfs-zimwriterfs.o] Fehler 1
    

    .
    What do I have to do to make magic.h acquaintant to zimwriterfs?

    Thank

    frank

     
  • franc walter
    franc walter
    2014-02-07

    OK, I downloaded (cloned) libmagic from
    https://github.com/glensc/file.git
    did:

    autoconf
    automake
    autoreconf -i
    ./configure
    make
    make install
    

    and somehow it did compile. I have to read about the GNU Autotools, this is clear :(

    After this, I had to follow the hint of Dmitriy in this thread:
    https://sourceforge.net/p/kiwix/discussion/604122/thread/bc4b6d0f
    ("new zimwriterfs : libzim.so.0 No such file or directory")

    and I did:

    $cp /usr/local/lib/libzim* /usr/lib
    

    and I can start zimwriterfs without errors :)

    Now I am only missing a bit something like a howto of this file, because the "help"-file is not very verbous:

    # ./zimwriterfs
    You have too few arguments!
    zimwriterfs --welcome=html/index.html --favicon=media/favicon.png --language=fra --title=foobar --description=mydescription --creator=Wikipedia --publisher=Kiwix [--minChunkSize=1024] DIRECTORY ZIM
        DIRECTORY is the path of the directory containing the HTML pages you want to put in the ZIM file,
        ZIM       is the path of the ZIM file you want to obtain.
    

    .
    But I will try to struggle trough it, without nearly any support like I did before :/
    I will do it at least once, then I will compare it to Xowa, maybe there is more future for an offline wiki for smartphones there...

     
  • franc walter
    franc walter
    2014-02-07

    ./zimwriterfs --welcome=html/index.html --favicon=media/favicon.png --language=de --title=dewiki --description=Wikipedia-ohne-Bilder --creator=Wikipedia --publisher=Kiwix --minChunkSize=1024 /home/franc/kiwix/static /home/franc/kiwix/dewiki.zim
    
     
    • Dmitriy
      Dmitriy
      2014-02-07

      You can go to drink tea now - two russian "samovars" should be enough for DEwiki :)

       
      • franc walter
        franc walter
        2014-02-07

        :) drank coffee and beer instead.

        But now I came back and discovered that I ran into an error, zimwriterfs stopped:

        ...
        ...
        sort 2986877 directory entries (url)
        set index
        translate redirect aid to index
        internal error: redirect aid html/R/e/d/_/Red_Bull_Ring.html not found
        

        .
        Maybe that Red_Bull_Ring.html is really not existing, I haven't even checked, but anyway, can I set it up, that it will just continue to pack and ignore missing article-files?
        I guess there are uncountable articles missing, under some millions they very likely don't count (Red Bull Ring e.g. seems not very interesting), but I am sure, I cannot find any missing articles, so if zimwriterfs just stopps I don't see any chance to complete the zim file :(

        Any idea?

        Thank!

        PS.: now I started like this:

        root@me:/home/franc/kiwix/zimwriterfs# ./zimwriterfs --welcome=html/index.html --favicon=media/favicon.png --language=de --title=dewiki --description=Wikipedia-ohne-Bilder --creator=Wikipedia --publisher=Kiwix /home/franc/kiwix/static dewiki.zim > /home/franc/kiwix/zimwriterfs/log.txt
        

        .
        where log.txt is the output which I show (in another screen-session) with:

        tail -n 50 -f log.txt
        

        So I don't miss any output, which is truncated in screen-sessions.

         
        Last edit: franc walter 2014-02-07
  • franc walter
    franc walter
    2014-02-07

    Now I checked, and your right! No, not missing!
    File is there!
    So what?
    Amazing. Don't see any permission-problems, file has 644, can open it.
    But when I open the file, I can read:
    ...
    URL=../../../../../html/R/e/d/_/Red_Bull-Ring.html
    ...
    .
    So this file is definitely missing, there is no Red_Bull-Ring.html in this folder.

    Is there any way to tell zimwriterfs to jump over not existing files?

     
  • franc walter
    franc walter
    2014-02-07

    Again this error:

    internal error: redirect aid html/R/e/d/_/Red_Bull_Ring.html not found
    

    .
    I think I must look into this (buggy?) code :(

    EDIT: First I tried it with deleting the meta-tag with the redirect to this not existing file.
    But I am not very confident, that I will just jump into the next missing file.
    EDIT2: till yet it is running, dewiki.tmp has 300 MB at the moment, in 9 hours.

     
    Last edit: franc walter 2014-02-08

    • Anonymous
      2014-02-08

      I never have met this problem. Would be great if you could isolate the problemen an share the steps in a bug http://reportabug.kiwix.org

       
  • franc walter
    franc walter
    2014-02-08

    Hurray, 17:09 ZIM-File is ready!!!
    Only this one file was corrupt or something.
    here you can download this (yet untested) zim of german dewiki:

    http://7fw.de/tmp/dewiki.zim

    I cannot test it right now, only tommorrow, so only then can say, if it works, but I got at the end:

    Packing data for /M/Title
    186977 clusters created
    fill header
    write zimfile
    ready
    

    .
    So I think must work :)

    Thank for help, Dmitriy!!!

    frank

    EDIT: as this problem was only with one missing file (Red_Bull-Ring.html) I guess it would be a mwoffliner.js issue, which seems not really stable to me.
    I don't know if maybe now with redis it is stable, because I started without when I had all these errors and had the most downloaded already when using the new version with redis, but I will test this next time.

     


Anonymous


Cancel   Add attachments