It is intuitive to be able to call "<fail/>" within the "onsuccess" task. At this time, you cannot, the build succeeds regardless of what you do within "onsuccess".
I suggest adding a property "nant.onsuccess.allowexceptions" (or similar). By default, it's false. If it's set to "true", calling <fail/> within "onsuccess" will cause the build to fail.
See also this (old) bug: http://sourceforge.net/tracker/?func=detail&atid=402868&aid=880616&group_id=31650