Because there are no torques acting on the book, the (vector) angular momentum does not change, but the (vector) angular velocity does change, and the visual effect is that the book tumbles in an unexpected way.
You can see this effect in the real world. Try flipping a book or a tennis racket in the air. You'll find that if the object rotates around its long axis or the axis perpendicular to the largest side, the rotation is stable. If however you flip it around the third axis, no matter how carefully you try to flip it exactly around that axis, you'll see that the object tumbles.
P.S. Steve Spicklemire and I believe that we're very close to the next release of the wxPython-based VPython 6. All that remains is some documentation and some cleanups of keyboard issues.