I have Webmin 1.580 set up on a Debian 6 box and set it to use the German language. The group enumeration is done via "compat winbind" in /etc/nsswitch.conf, winbindd asking a german Windows server 2003 R2. A call to "getent group domänen-admins" yields:
# getent group domänen-admins
So, we do have group names with umlauts.
As in Webmin the default encoding for language 'de' is iso-8859-1 unfortunately, with winbindd giving out utf-8 coded strings, the output of group_chooser.cgi is not really usable. Groups without umlauts are recognized and can be saved, but chosing groups with umlauts results results in unusable entries in for example the Samba module, nor do the members of that groups enumerate in group_chooser.cgi.
Most probably this happens in user_chooser.cgi, too - but we have no usernames with umlauts gladly, so I did not check that.
I attached a fairly ugly patch to fix our very situation. It converts the group names from utf-8 to iso-8859-1 using the "Encode" module, regardless of which language you have set in Webmin. As we do only use it in German, it works for us, but is for sure way to unflexible to work elsewhere, I guess.
With kind regards,