If AltGr is down, leave it down, the translators list
when it needs to be added not removed.
It's either already held for the key (if the keyboard mappings are
the same), or it isn't and isn't required for that keysym.
Currently for en-us keyboard layouts the right alt is nearly
useless. It can't be used with any other key in chording
such as Alt-F4, Alt-F, Alt-Left, Alt-Right, trying will
send an Alt_R up before and Alt_R down after causing windows to
see it as if the Alt_R wasn't held.
I think this is a good compromise, those who are using Alt_R for
AltGr composition will be used to not holding it when pressing
other keys, and for the rest of us it will finally work.
Note, I don't have an AltGr client or server to test this with.