Menu
▾
▴
kiwix-developer
|
From: Adrian C. <Adr...@em...> - 2014-09-15 07:22:44
|
After successfully building a ZIM file from a subset of the HTML rendering of our wiki, I put zimwriterfs to build a ZIM file from the whole wiki content. It is actually the content of three different wikis, which I wanted to build into a single ZIM file (since Kiwix does not support inter-ZIM linking, and we have interwiki links between these wikis). I started the process at 9:30 on Friday, and when I left at 18:00 the build process was at: ... Creating entry for favicon.ico sort 118267 directory entries (aid) remove invalid redirects from 118267 directory entries sort 118267 directory entries (url) set index translate redirect aid to index 118267 directory entries created create title index 118267 title index created create clusters 0% ready Packing data for examples/IE80Fixes.css Packing data for examples/IEFixes.js Packing data for examples/ajax.js Packing data for examples/commonContent.css Packing data for examples/commonElements.css Packing data for examples/commonInterface.css Packing data for examples/commonPrint.css Packing data for examples/config-cc.css Packing data for examples/config.css Packing data for examples/config.js Packing data for examples/feed.css Packing data for examples/geshi.css Packing data for examples/jquery-latest.js Packing data for examples/jquery.cookie.js Packing data for examples/jquery.delphionly.js Packing data for examples/jquery.fixedtableheader.js Packing data for examples/jquery.protected.js Packing data for examples/jquery.switch.js Packing data for examples/main.css At 9:00 today, Monday, the executable was still at the very same point, so I stopped the executable. On Friday I noticed that the zimwriterfs process was not idle, it just was at 0% processing most of the time but it regularly peaked, so I assumed that it was just going page by page. Now, I am not sure, it looks more like it got stuck somehow. So, I have two questions that I hope someone can answer for me: * With that many entries (118267), is this normal? Or did the program got stuck? How much time does it take to do this for a Wikipedia? * Is there any trick that I can use to reduce the build time (e.g. to no more than 4 hours)? If it got stuck, any suggestions on how to debug it? CONFIDENTIALITY NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. |
|
From: Emmanuel E. <ke...@ki...> - 2014-09-15 07:39:07
|
Hi Adrian This is not normal at all. zimwriterfs is pretty fast and within a few dozen of minutes, the ZIM file should be ready. Regards Emmanuel On 15.09.2014 09:22, Adrian Chaves wrote: > After successfully building a ZIM file from a subset of the HTML > rendering of our wiki, I put zimwriterfs to build a ZIM file from the > whole wiki content. It is actually the content of three different wikis, > which I wanted to build into a single ZIM file (since Kiwix does not > support inter-ZIM linking, and we have interwiki links between these > wikis). > > I started the process at 9:30 on Friday, and when I left at 18:00 the > build process was at: > > … > > Creating entry for favicon.ico > > sort 118267 directory entries (aid) > > remove invalid redirects from 118267 directory entries > > sort 118267 directory entries (url) > > set index > > translate redirect aid to index > > 118267 directory entries created > > create title index > > 118267 title index created > > create clusters > > 0% ready > > Packing data for examples/IE80Fixes.css > > Packing data for examples/IEFixes.js > > Packing data for examples/ajax.js > > Packing data for examples/commonContent.css > > Packing data for examples/commonElements.css > > Packing data for examples/commonInterface.css > > Packing data for examples/commonPrint.css > > Packing data for examples/config-cc.css > > Packing data for examples/config.css > > Packing data for examples/config.js > > Packing data for examples/feed.css > > Packing data for examples/geshi.css > > Packing data for examples/jquery-latest.js > > Packing data for examples/jquery.cookie.js > > Packing data for examples/jquery.delphionly.js > > Packing data for examples/jquery.fixedtableheader.js > > Packing data for examples/jquery.protected.js > > Packing data for examples/jquery.switch.js > > Packing data for examples/main.css > > At 9:00 today, Monday, the executable was still at the very same point, > so I stopped the executable. > > On Friday I noticed that the zimwriterfs process was not idle, it just > was at 0% processing most of the time but it regularly peaked, so I > assumed that it was just going page by page. Now, I am not sure, it > looks more like it got stuck somehow. > > So, I have two questions that I hope someone can answer for me: > > ·With that many entries (118267), is this normal? Or did the program got > stuck? How much time does it take to do this for a Wikipedia? > > ·Is there any trick that I can use to reduce the build time (e.g. to no > more than 4 hours)? If it got stuck, any suggestions on how to debug it? > > > CONFIDENTIALITY NOTICE: This email message is for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized review, use, disclosure or distribution is > prohibited. If you are not the intended recipient, please contact the > sender by reply email and destroy all copies of the original message. > > > ------------------------------------------------------------------------------ > Want excitement? > Manually upgrade your production database. > When you want reliability, choose Perforce > Perforce version control. Predictably reliable. > http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Kiwix-developer mailing list > Kiw...@li... > https://lists.sourceforge.net/lists/listinfo/kiwix-developer > -- Kiwix - Wikipedia Offline & more * Web: http://www.kiwix.org * Twitter: https://twitter.com/KiwixOffline * more: http://www.kiwix.org/wiki/Communication |
|
From: Adrian C. <Adr...@em...> - 2014-09-15 08:57:14
|
Thanks. After your answer, I decided to debug the issue, I found out what the cause was and opened a bug report: https://sourceforge.net/p/kiwix/bugs/740/ -----Mensaje original----- De: Emmanuel Engelhart [mailto:ke...@ki...] Enviado el: lunes, 15 de septiembre de 2014 9:39 Para: Adrian Chaves; kiw...@li... Asunto: Re: [Kiwix-developer] How long is it normal for zimwriterfs to spend on 118267 entries? Hi Adrian This is not normal at all. zimwriterfs is pretty fast and within a few dozen of minutes, the ZIM file should be ready. Regards Emmanuel CONFIDENTIALITY NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. |
|
From: Emmanuel E. <ke...@ki...> - 2014-09-19 12:59:12
|
Hi Adrian Great, hope you finally have achieved to get a working ZIM file of your content. Regards Emmanuel On 15.09.2014 10:57, Adrian Chaves wrote: > Thanks. After your answer, I decided to debug the issue, I found out what the cause was and opened a bug report: https://sourceforge.net/p/kiwix/bugs/740/ > > -----Mensaje original----- > De: Emmanuel Engelhart [mailto:ke...@ki...] > Enviado el: lunes, 15 de septiembre de 2014 9:39 > Para: Adrian Chaves; kiw...@li... > Asunto: Re: [Kiwix-developer] How long is it normal for zimwriterfs to spend on 118267 entries? > > Hi Adrian > > This is not normal at all. zimwriterfs is pretty fast and within a few dozen of minutes, the ZIM file should be ready. > > Regards > Emmanuel > > CONFIDENTIALITY NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. > > ------------------------------------------------------------------------------ > Want excitement? > Manually upgrade your production database. > When you want reliability, choose Perforce > Perforce version control. Predictably reliable. > http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk > _______________________________________________ > Kiwix-developer mailing list > Kiw...@li... > https://lists.sourceforge.net/lists/listinfo/kiwix-developer > -- Kiwix - Wikipedia Offline & more * Web: http://www.kiwix.org * Twitter: https://twitter.com/KiwixOffline * more: http://www.kiwix.org/wiki/Communication |
Thanks for helping keep SourceForge clean.
X