SourceForge is proud to announce the February 2013 Project of the Month, Kiwix, an offline Wikipedia reader.
I recently spoke with Emmanuel Engelhart, one of the developers on the project.
Rich: Congratulations on winning the SourceForge Project of the Month for
February.
Emannuel: Thank you for hosting Kiwix development tools and promoting free software.
Rich: Start by telling us what Kiwix is. How would someone use this?
Emmanuel: Kiwix allows to read Wikipedia offline. In addition, using the highly efficient ZIM file format (http://www.openzim.org), Kiwix can read any HTML content offline. In order to enjoy Wikipedia offline, you need to download Kiwix and a ZIM file of Wikipedia (from the Kiwix web-site or directly from the Kiwix internal library).
Then you can surf in Wikipedia as if you were online. Kiwix provides almost everything you will need:
- Case and diacritics insensitive full text search engine
- Bookmarks & Notes
- ZIM based HTTP server
- PDF/HTML export
- Localized in more than 80 languages
- Search suggestions
- Tabs navigation
- Integrated content manager/downloader
Rich: How did you come to start this project?
Emmanuel: Why lock up Wikipedia to Wikipedia.org? The contents of Wikipedia should be available for everyone! Even without Internet access. This is why we have launched the Kiwix project.
Rich: Can you give us some examples of your project being used in the real world?
Emmanuel: Our users are spread all over the world: sailors on the oceans, poor students thirsty for knowledge, globetrotters almost living in planes, world’s citizens suffering from censorship or free minded prisoners. For all these people, Kiwix provides a simple and practical solution to ponder about the world.
Kiwix is used,for example, by the Wikimedia France Afripedia project, and also by
Wikimedia Kenya. And in India.
Spreading work is done by Wikimedia people and by third parties like NGOs. A lot of individuals also download Wikipedia offline once and then share it with their friends and relatives. We have had around 100.000 downloads in January.
Rich: Release more and more up-to-date content is our top priority. We continuously increase our ZIM file throughput by improving our ZIM generation toolchain. We will also soon start to release offline version of other Wikimedia projects.
To make Kiwix work on smartphones is our second priority. We hope to release a first version of kiwix-mobile for Android in April.
Regarding the far future, we will try to be one of the best open-source e-book readers. We think we have a cutting-edge file format with ZIM which is perfectly complementary with the actual EPUB standard. We will do our best to offer the best user experience with both in the future.
Rich: If someone wanted to get involved in your community, what could they do? Are you looking for developers? Translators? Users? Testers?
Emmanuel: Actually, the most important work to do, can be done by everyone: this is promoting and sharing Wikipedia offline with Kiwix. We have remarked that most of the people, although they would really need it, think it is impossible to have the whole Wikipedia with pictures on a USB stick. That’s why we need people to setup projects and spread it.
But, we have also plenty of work otherwise, for example:
- We would love to have a new javascript developer able to add the EPUB support using Monocle
- We always need new translators for the user interface
- We are working currently on a solution for plug computers called kiwix-plug; and there is a lot to do for people having GNU/Linux admin skills:
- We need a C++ expert to help us improving tools to manipulate ZIM file, we need especially a solution to make ZIM incremental updates.
- Of course need we testers, join us at https://lists.sourceforge.net/lists/listinfo/kiwix-testing
- We need a bug master to manage all the tickets on the bug tracker
If you are interested, simply join us on Freenode IRC #kiwix channel.
Félicitations Emannuel, bien sûr que nous soutenons ton projet.
u know i am from algeria so this kiwix we traying it in 2022 or more lol ,
i ask u if u try it , ur idea about it !!!
From the first sentence I was wonderig about “how to they manage incremental updates” – something, that would be the first thing to find a solution for before even starting… the “job offers” section at the end of the interview then reveals: they did not think about it. Can this be true? It simple does not make sense rolling out a new file format without thinking of the key problems BEFORE! But why exactly does rsync not work for this?
OK, taking a quick look at the specs it seems clear, that rsync is not a good solution, because the file format is a compressed one and therefore we have for one new single character a completely new file.I thought this was a technique of the 90s where “distributed” still was something unknown to many programmers.
Looking at openzim.org website I also can not find one single sentence about synchronization and updating – the concept of git, however, does not seem to be unknown to the developers, as they are using it for their own code, so I wonder, how they did not make the little jump to understand that decentralised distribution of content might have something to do with differential updates…
So it seems to be true: they did invent a file format for distribution of content without thinking about how to update. In fact on the kiwix website you find one 10GB file for the 2013-01 Wikipedia – so in february one million users will have to download again 10,5 GB?
I know ranting is evil, but I am playing this role as a sacrifice for the open source community – we have to speak clearly about such kind of extremely dumb decisions to avoid hundreds of programmers making the same error ever again. There is no sense in ignoring this elephant – which is a mammouth in fact! The inventors of this file-format should get the price of the “dumbest-programmig decision-of -the month”, nothing else. Doing content distribution without thinking about a sane solution for updates is an unbelievable short-sighted WTF.
To be at least a little bit constructive:
– what parts of git can be used to help these guys? Or:
– why is git not good for them?
Can kiwix, instead of using a braindead not-update-enabled fileformat, be re-build as a git repository enabled wikipedia-markup interpreter / reader? If not, why not?
Using git as a “file backend”, transparent for the user, the update problem could be solved very quickly. Also the outcome might be something, that is usable for many more things, if we keep the markup-interpreter part pluggable.
Forks welcome, Good Luck!
Félicitations bien sûr que nous soutenons ton projet.
[…] that data more accessible to people worldwide. Emmanuel Engelhart, one of the project developers, recently sat down with SourceForge’s Rich Bowen to talk about Kiwix’s development and what it still needs from anyone willing to help out. […]